ฝัง
- เผยแพร่เมื่อ 30 เม.ย. 2024
- Follow Along: console.brev.dev/launchable/d...
Join Our Discord: / discord
In this guide, we fine tune the popular open sourced model, LLaVA (Large Language-and-Vision Assistant) on a dataset to be used in a visual classification application. You can perform the fine tuning yourself, regardless your level of experience, or the level of compute you have access to.
Please leave any future guides you would like made below!
![[SUB] 베몬이 교관님 멘탈을 탕탕 후루후루~٩(๑❛ᴗ❛๑)۶ FOREVER 웃게 되는 갓기들의 수련회 | 돌들의침묵 | BABYMONSTER(베이비몬스터) 'FOREVER'](http://i.ytimg.com/vi/FISAO7BwIxE/mqdefault.jpg)
You know bro is a A level engineer when he can explain stuff soo easily
Big man ting yeh. Looking good brev
Best guide/insights on fine tuning I’ve seen. Subscribed 🔥
Baxate, you’re the goat. For a beginner like myself, that was a very useful video
glad you found it useful!
I tried out brev for my machine learning course, love the options for the payment system where i have the option to cut if off after X dollars, low prices, and ui looks awesome. I know it said it somewhere but it took me a minute to realize that my Jupyter notebook takes around 4 minutes to launch so for blind ppl like me I’d put some more text saying Jupyter notebook will be created in X minutes.
Love this vid and outreach- I’ll keep watching Baxate
Thank you so much for the kind comment! I will bring that feedback to the team :).
Very useful! Subscribed indeed 🙂
Thank you, this is great
bro the goat
Bax ate with this one!
yurrr
man you are awesome!
Thank you
twin served meat 🔥🔥
Great video:) Can you please comment on the dataset size? The one you used consists of roughly 9k samples. How many samples are needed to have a decent lora fine-tune? I've heard that with LLMs you can achieve much even with only a few examples. Is it the case for LLava as well? Please share any more information you can on the dataset creation. thanks!
Hello bro , after running the deepspeed script there is no file with name mm_projector.bin is generated which is required in merging process but a non_lora_trainable.bin is generated
is it possible that the model can tell you there a picture was taken(geographic), based on probability, and purely focuses on this, because you give him the information in fintune( im a beginner)
Cool demo, thank you. Could you share some examples of training data? That new model is great. Can you share it on Hugingface? How big did it end up being for inference purposes,
Hey Paul! Here is the documentation for the model on hugging face:
huggingface.co/docs/transformers/en/model_doc/llava
Here is the training dataset:
huggingface.co/datasets/Multimodal-Fatima/OK-VQA_train
Here is the testing dataset:
huggingface.co/datasets/Multimodal-Fatima/OK-VQA_test
Note that we did not create the model, nor the training or testing datasets! We are simply using them as an example here
Came from IG
Hey, I had a query regarding generating the custom dataset using gpt 4, shown at the very beginning. It seems it does not generate json file with the exact format necessary for LLaVA
came from ig
For this use case, why didn't you just use prompt engineering (using a very specific prompt) to give you the same output?
Came from TikTok! But I have no experience w AIs but am surely going to dive in to train a model for my startup application. Do you think this model could be trained to estimate macros from an image, let’s say in buckets or ranges, after identifying the food itself?
yes, absolutely! That is a perfect use case
Can you show how to fine-tune VILA models from Nvidia?
Have you used this link? I'm reporting an error when loading the dataset now, if you can please take a look . thank you
Wouldn't prompting the LLM in various scenarios in the application code be enough to get the right response? I am not clear on fine-tuning.
@brev-dev Yeah I feel so too. Finetuning is more useful when you have a novel category or more deeper classification. For example general model identifies the dog, but the fine-tuned identify the breed as well. Then you need to have the tagged dataset of dog breeds.
Wouldn't it have been simpler to feed the fluffy text to llama3 to come up with the summary?
Do I have to buy credits to follow along?
i see you're finetuning LLaVA 1.5 is it possible to use this notebook for 1.6 too?
baxate !
Is this video for elementary school ?
In what world is this a “beginner friendly machine learning guide”? What💀💀💀😂😂
let me know where you struggled! I tried to explain the concepts at a high level and run the cells as they were written.
@@brev-dev thanks for replying :)
Carters insta story said “beginners guide” so i thought it would be a intro to machine learning or something
But after seeing this video …
A beginner WOULD NEVER be able to comprehend a single sentence in this video 😂😂
@@aamirshaikh2100 This is Carter :). I will keep that in mind and maybe make a dedicated intro to machine learning video!
@@brev-dev thanks for taking the time 💓
@@aamirshaikh2100 I am not Carter (or in any way related to the channel, lol) but if you tell me what is your starting point I can send you some resources or more specific questions, I'm not a pro yet but it may be useful if you're coming from zero