The EASIEST way to finetune LLAMA-v2 on local machine!
ฝัง
- เผยแพร่เมื่อ 26 ก.ย. 2024
- In this video, I'll show you the easiest, simplest and fastest way to fine tune llama-v2 on your local machine for a custom dataset! You can also use the tutorial to train/finetune any other Large Language Model (LLM). In this tutorial, we will be using autotrain-advanced.
AutoTrain Advanced github repo: github.com/hug...
Steps:
Install autotrain-advanced using pip:
- pip install autotrain-advanced
Setup (optional, required on google colab):
- autotrain setup --update-torch
Train:
autotrain llm --train --project_name my-llm --model meta-llama/Llama-2-7b-hf --data_path . --use_peft --use_int4 --learning_rate 2e-4 --train_batch_size 12 --num_train_epochs 3 --trainer sft
If you are on free version of colab, use this model instead: huggingface.co.... This is a smaller sharded version of llama-2-7b-hf by meta.
Please subscribe and like the video to help me keep motivated to make awesome videos like this one. :)
My book, Approaching (Almost) Any Machine Learning problem, is available for free here: bit.ly/approac...
Follow me on:
Twitter: / abhi1thakur
LinkedIn: / abhi1thakur
Kaggle: kaggle.com/abh...
Anyone comes across this in 2024 (jan ), the command switches with new autotrain version is autotrain llm --train --project-name josh-ops --model mistralai/Mistral-7B-Instruct-v0.2 --data-path . --use-peft --quantization int4 --lr 2e-4 --train-batch-size 12 --epochs 3 --trainer sft . Great, Video, thanks Abhishek
After finetuning it, how to run it?
@@BrusnickiRobertoHey hi did you run it??
@@vinodb4339 no
Keeping it this simple is something very few people are able to do. Very well explained.
This can be understood by even a beginner. Atleast the execution if not the intuition behind it. Kudos
Amazing, tutorials at light speed! Llama 2 was just released! 😮
🙏🏽
😂😂Yeah exactly
This man is a genius lol
Great tool, it works for my small dataset (248 rows) and i trained it in a rtx 3060 with 12G :)
Appreciate it, and request to continue making such videos🎉
That's Awesome, nothing better than this way of training large language model. Super easy ❤
Incredibly helpful video, I appreciate that you took the time to create this! Great stuff
There is only one thing I want to see. I want to see you using the final result and prove it actually works. Thank you.
As always very useful and short without wasting anyone's time. Thank you. Just I'm a bit confused about the prompt formatting you have used here - "### Instruction:
### Input:... etc" while Llama official is "[INST] {{ system_prompt }}{{ user_message }} [/INST]" and on TheBloke's page it says "SYSTEM: {system_prompt}
USER: {prompt}
ASSISTANT:"
I think this isn't mandatory, it is a suggestion.
Thank you for the on-point tutorial.
Great tutorial! Can you also put up one video teaching on how to merge the fine tuned weights to the base model and do inference? Would like to see an end-to-end course. Thank you!
+1 on this question !!!!
any updates guys?? I really want to know how to merge the fine tuned model with the base model and do the inference. Do let me you have any resources or insights about the same
@@devyanshrastogi also need this answered. have you guys had any success?
Thank you Abhishek! This is phenomenal.
Astonishing content Man 🔥🔥🔥 🚀
Can you make a video for fine tuning in silicon macs ?
Great video! Would be great if in some future vid you could go into depth on the training hyperparameters and perhaps also talk about what size your custom datasets should be.
sometimes I do that. however, this model would have taken wayy too long to train. im training a model as i type here and if i get good results ill share both model and params 🙂
@@abhishekkrthakur guess no good luck with the training :(
Thanks for the tutorial! A couple questions for you. Is there an approach you're using to test quality and verity that the training data has influenced the weights in the model sufficiently to learn the new task? And second, can you use the same approach for unstructured training data such as using a large corpus of private data to do domain adaptation?
Hi Abhishek, is the auto train using LORA or prompt tuning as the PEFT technique?
In my experiment, it not create the [config.json] what am I doing wrong?
I trained model using autotrain in same way as you suggested and model file is stored.
Now I need to use this model for prediction. Can you shed some light on this as well?
Hello I am getting this error can someone please help me out with it: ValueError: Batch does not contain any data (`None`). At the end of all iterable data available before expected stop iteration.
what kind of GPUs do you have? how big was your dataset and how long did it take to train? what is the smallest fine-tuning data set size that would be reasonable?
I know it could be too much but could you also make a video of how to prepare the data? I have like 'question' and 'answer' but I am strugging to make it to a trainable data set into that kind of csv so I could use it!
The way Abhishek side eyes before stopping the video and resuming is is soo crazy 🤣🤣😅
lol. big screen. button too far 🤣
Informative video, hopefully one day I will get a task that requires me to finetune a LLM
or you can just do it for fun 🤗
Excellent, thank you so much. I will try.
Why do we need --mixed-precsion and --quantization both? Aren't they both doing the same thing?
Please show how you used the trained mode for inference
Yes. Please!
Thanks Kaggle grand master :) you got an channel.
Nice video Abhishek!
I am curious to know about custom data for LLMs. What is the ideal (good quality) data size (e.g., #rows), to fine-tune these models for good performance, not necessarily it should be big data of course.
Thanks!
One other comment. The data you showed looks randomly organized and I don't understand how it's a CSV at all. The top part has some column names and data below is not following it at all. What is going on?
On text column you formatted the data as ### Instruction: and ### Response:
Some use ### Human and ### Assistant :
How will autotrain will understand these custom formats?? Also how will it decide to read train.csv when we just --data_path . ???
doesnt matter what format you use as long as you use the same format during inference
@@abhishekkrthakurty for the reply. Can you show/link this inference setup from any of your videos / our from others' ?
Do let me know if you found any update regarding the same
Great one! Two things - you didn't show how to configure the cli to enable access to the model. Secondly, it would be useful to know how to use aws for training. Thanks!
Hi I have a question , it´s posible to use "autotrain" without cuda, with rocm support of AMD GPU ?
Thanks for sharing!
For fine-tuning of the large language models (llama-2-13b-chat), what should be the format(.text/.json/.csv) and structure (like should be an excel or docs file or prompt and response or instruction and output) of the training dataset? And also how to prepare or organise the tabular dataset for training purpose?
Can I try your code on Mac M2 ( 64GB memory)? Thank you!
even I was looking for Mac M2, I have only 24G
Any idea to fine-tune it on Apple chip M1/M2? Just have an installation issues with auto train-advanced 😢
Would love to know how to do this as well!
Bumping because running into so many issues with M1. Cannot believe how little resources are available for M1 right now given that macOS is so widely used in data science
Great work Abhishek Thakur, it will be great if you made a video on how to run the fine-tuned model
already done. check out other videos on my channel
@@abhishekkrthakur can you please post the video link? there are many videos in your channel, it's hard to find which one you are talking about.
@@abhishekkrthakur I did fine tuning on the model, but I don't think I can run it on google colab with T4 since its show out of memory error!! Any suggestion?
@@AnandMoorthyJ did you find the video? im looking for the link aswell :)
@@AnandMoorthyJare you able to run it??
Did you find the video link?
Great Video. i love it and can't wait to try it. Now that Llama2 is out... is it better to FineTune a model or try to create your own Model?
How do you convert it to work with Ollama? I setup the model file and it doesnt seem to know anything from my training.
When will I be able to see the model that I have trained? Once all the training is complete?
Do you have to use [INST]...[/INST] for indicating the instructions? I think the original Llama 2 model was trained with these tags, so I am a bit puzzled if you have to use the tags in the csv or they are added internally ?!
in this video, im finetuning the base model. you can finetune it anyway you want. you can even take the chat model and finetune it this way. if you are using a different format for finetuning, you must use the same format while inference in order to get the best results.
Hi Abhishek,
Much appreciated. How would things change if we were to use simple fine tuning? That is, just a large single code file to learn from, to tune code-llama, phi2, etc..
thanks for sharing!
Hey Abhishek, Thanks for sharing this amazing tutorial. Can I do this on my RTX 3070Ti 8GB GPU? If yes, what batch size would be preferable?
8GB sounds a bit low for this. maybe try bs=1 or 2? but tbh, im not sure if it will work. Might work fine for a smaller model!
nice video. it great for a beginner to learn how to fine tune LLAMA2 locally. It would be better if you can share the code and dataset.
Got it to work in Ubuntu WSL:
Have Conda installed..
1 apt-get update && apt-get upgrade -y
2 conda update --all
3 apt install nvidia-cuda-toolkit
4 conda create -n train python==3.10
5 conda activate train
6 pip install autotrain-advanced
7 autotrain setup --update-torch
8 conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.7 -c pytorch -c nvidia
9 apt install build-essential
10 pip install cchardet
First thank you for the great tutorial , I have one question : I am trying to finetune the model on Japanese , do you have any advice for that ? I have tried the same script as you did but it didn't work; it produced some gibberish after the training finished , I am guessing it is a tokenizer problem, what do you think ?
How do you set the training data? I see different people using different formats? Does it matter or is the only requirement that it has to be structured meaniningfully?
Hi Abh
Your method is inspiring and commendable. How do we read the csv or json training dataset we prepared instead of the hugging face dataset you used?
I am getting a "No GPU found. Please install CUDA and try again." Even after Installing CUDA I am getting this error. When I check with nvcc -V I get NVIDIA Cuda Compiler Driver message. Is there an issue with the path?
did you come across a solution @ravigarimella3166
thank you very much
If there are a multiple input values how that have to be inserted in the cvs data ?
Great Job :)
Very good tutorial. On what machine are you running this? I am trying to run it on a Macbook pro M1 but I keep getting "ValueError: No GPU found. Please install CUDA and try again." I have tried to do this directly on Huggingface and got "error 400: bad request"...so I cloned autotrain and ran it locally...still getting error 400. Do you have any pointers?
Same error
Nice vid
Should we label input tokens to -100 to focus the training on the prediction?
I see no one doing it
I just bought a RTX 4090 Founders Edition. Could you tell on a particular example were I could run into limits with card when training LLMs locally? I personally think that I'm safe for the next few years and I will not run in any problems.
I did the same with LLama-2, but --push_to_hub doesn't push at all.
this is really great appreciate it, but i always face problem in installations, like i tried to install pip install autotrain-advanced and i am getting error aas could not build wheels for pycocotools , here are the steps i performed i created a new environment and installed pandas and then this autotrain-advanced installation .. and its failing, if you can help that will be great.. thanks in advance
I found the solution and was able to move forward, but now when i am trying to train it is saying that it is a gated repo and needs permission, there was a form and i submitted the data, any idea by when i will be getting the permission, do they give to everyone or what is the possibility here? also can i do the same with ollama otherwise? as i feel it might be easy with ollama.. kindly help here. thanks in advnace.
Amazing content. One Q left: how can I run the model locally in inference mode after training? Anyone have a command for that?
th-cam.com/video/o1BCq1KJULM/w-d-xo.html
Thanks for the tutorial.
On a Linux 5.15.0-71-generic #78-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux machine, I get following error when training llm with the small dataset. File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2819, in from_pretrained
raise ValueError(
ValueError:
Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
the quantized model. If you want to dispatch the model on the CPU or the disk while keeping
these modules in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom
`device_map` to `from_pretrained`.
What could be the problem? Is it possible to share the data.csv that you have with single row that I can take as reference to test my own data?
Thanks for the video. i have a doubt that how can i make a package of it so others can also use it offline by just installing the application
hello, thank you. i really need to know: does this pad appropriately? also, how does it internally split it into prompt completion? Can i make up roles like ### System? does it complete only the last message?
How is your ipython giving suggestions? I want the same set up. Please make a video on these types of set up that assists in coding and other processes.
Any chance you have the python scripts available somewhere? Thanks in advance.
where is the load dataset part in ur code and in the jupyter notebook? did u even bother testing it b4 publishing?
Thank you ,what is diff between instruction and input
Hello,
Thank you for this video.
Can we run this on mac, though?
I ask because my training data is confidential and I don't want to risk it with cloud. I have access to a m2 ultra mac studio though
Thank you for this interesting video. How could we change the encoding to ''latin-1' in order to train on french language ? thank you.
Hey, could you guide us how to train custom dataset on llama2? How to prepare our dataset for training?
Hi when we tried fine tuning MPT LLM using autotrain, getting the error ValueError: MPTForCausalLM does not support gradient checkpointing. Any help you can offer on this pleas?
Beautiful content, I have a side question, what tool you are using to have "copilot"-like suggestion in your terminal? Thx again for the video
I use Hugginfaces co pilot. - it works pretty well and super easy to set up and free..
@@jessem2176 Thanks for the recommendation, but did you mean HuggingFace Personal-copilot Blog?
when i am using this command pip install autotrain-advanced getting errors
error: subprocess-exited-with-error
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
Thank you for the video. I am new to this, so I am trying to set it up on my windows PC. When I am trying to install the latest version of autotrain-advanced==0.6.2, I get an error saying: trition==2.0.0.post1 cannot be found. Which I believe is only available on Linux. So is it possible to use autotrain-advanced on windows?
Thanks for this great explanation. Can you provide the link to you training data?
which machine is recommended for fine-tuning LLAMA? windows?
Nice job!
Superb video! Does autotrain allow to export finetuned model as GGML file? Or can it be used with GGML file?
Where can i find to code you worte and your dataset? I would like to use it as an exemple for testing
Thanks for the comprehensive tutorial. Can this be done using chat logs to build a clone of your friend? I have done this with GPT3.5 finetuning using prompt->response. The prompts are questions generated by ChatGPT based on the chat log message. Can the same thing be done with Instruction->Input->Response? Thank you very much man.
thanks bro ❤
Hi , nice tutorial, could you please help me with this error ? , when I try to train the model I got this error : raise ValueError("No GPU found. Please install CUDA and try again.") , Do you know how to solve this ?
"These days", LOL. We should all be training our own LLMs.
Thanks!
Deceptive use of the term “fine tuning”. What you’re doing here isn’t fine tuning. It’s “many-shot” learning in the prompt.
I thought that too from the title as fine tuning required very powerful hardware... What this person did is in my opinion instruction fine tuning. Please share your thoughts I love discussion
correct me if im wrong but from what i understand, this is fine-tuning as we are using an entirely different dataset and have re-formatted the instructions, training the model to perform chat task and not classical text generation (which the base model is supposed to perform). many-shot / few-shot is when you use very little number of samples and expect the model to perform variety of tasks.
The base model is trained on a dataset and task that is entirely different than what we are training it to do. You might say its the same task: text-generation, but since the dataset is different, it is finetuning. Its same as taking a model such as ViT which is trained on images of different categories and training it to perform detection of pneumothorax given chest xray images. Base task remains the same but since dataset is different, its fine-tuning. many-shot/few-shot will involve using the model as it is on a entirely different task by showing it only a few samples: 10-20. This is similar to using a text generation model and showing it only a few samples for example a few pairs of english-german text pairs and then asking it to translate a user-provided english text in to german.
@@abhishekkrthakur That's correct. I'm working on a project. I was wondering if you could provide insights. I'm working on a blog generation, I've used gpt-3.5-turbo. I prompt engineered it minutely and it provided quite good results. I want to know if I know llama v2 for it as well, if fine tuning it would produce good results as it has 70B parameters compared to 157B parameters for gpt-3.5-turbo. Thank you!
I think you need knowledge based more than fine tuning because in LLM finetuning is for sounds and images but for content based we need knowledge
Great tutorial! Thank you! Maybe I missed it but, at which point do you enter your HF token?
You dont. You login using "huggingface-cli login" command. There's also a similar command for notebooks and colab. :)
I have llm downloaded can I fine tune it with downloading from huggingface.
Thank you. Can you please also show us how to train on GPU #3 or GPU#1 or both GPU#1&3 but not in GPU #0 in a multi GPU machine?
CUDA_VISIBLE_DEVICES=0 autotrain llm --train ..... will run it on gpu 0
CUDA_VISIBLE_DEVICES=1,3 autotrain llm --train ..... will run it on gpu 1 and 3
When tying to use inference API for finetuned model I am getting 'error': "Could not load model XXXX/XXXX with any of the following classes: (,) How to resolve this ?
Thanks, This really helps. Could you help me clarify my requirement? I am planning to train the LLama v2 over a pdf documents and I want to ask questions about it. I am planning to create CSV text chunks of the pdf documents. So instead of ##instruction ##response, It will be a text context. As a prompt, I am planning to add " You are an agent and carefully understand the context and answer to the questions asked ##context: " then I will follow the auto train to train the model. Do you think this will work ?
Working on a similar use case. Let's connect ?
Even i am working on this. Have you guys found out any solution?
@@shameekm2146 not yet
@@rishipatel4652 yes, did yo figure out any solution?
maybe can try vector DB + LLM practice
Hey! Fantastic video, but i do not understand at all the division into smaller chunks of the model in order to work in free version of collab, can you explain it? Thank you!
chunks are loaded into ram first. since larger chunks didnt fit in ram with all the other stuff, i created a version with smaller shards :)
Thanks for the nice tutorial. Could you show how to perform inference the finetuned model?
yes. its in one of my previous videos :) thanks!
@@abhishekkrthakur Which one?
How can i load the model for inference if i stop training after 1 epoch out of 3 epochs.
Can we do this on a CPU? I am using a GGmL model.. please share your thoughts
I added push_to_hub command but it didnt push.. how do I use it to test the output?
Thank for the tutorial!
How can I fine-tune the language model with a domain-specific unlabeled dataset to improve performance on that specific domain? Is it effective to leave the instruction and input empty and only use domain-specific text for the output?
Hey, Can we connect. Even I need help on similar topic. We can discuss more ...
It's very useful for me , can you share you instruction dataset or just tell me where I can look for some small and good dataset , thank's again !
I already found some tiny dataset from huggingface , I will follow your step , that's interesting thing !
Is there a way to run it on CPU? Thanks sir, I love your work
can i train llama3 also with these steps?
Hello @abishekkrthakur can I train summarization models with autotrain advanced?