Thanks for the great content. A video going through the fine-tuning process on this one would be amazing. I am not sure how this could scale to a video implementation (probably passing a frame each time).
Thanks for your work on sharing this information. Much easier to watch your content than keep my ear to the ground all day trying to keep up. Much appreciated, sir.
@samwitteveenai please make a fine-tuning video about VLMs such as: Llava, Florence-2 and if possible try to use Ollama so that we can make the inference on local device.
I've tried this model, describing the image is great. I've also tried the docvqa, but giving only one word answers and not getting even simplest questions right. i had hoped to do some classification and compare with other models.
Thanks for the information this is great. Can i fine tune it for certain specific images like few short learning. Can you put a tutorial for the same it will be great full.
Would be interested on how much memory is required to run these models. they seem pretty small even unquantized. Maybe I will try it later on my 8GB M1 Mini. One thing I am curious about: at 3:38 , the description for the image is wrong in ways that seem odd. The title is described as being on top with the "20 Years of ..." underneath and Ron's tie is described as red and hair blonde. I wonder if this is just vagaries of the model (placement data would be strange) or over reliance on training data. Or a straight up mistake in 'creating' the paper (which would probably be the most disturbing😉).
I think fine-tuning for OCR would be a good demo. OCR in the real world with images of documents is much harder than OCR on electronic documents so would be cool to see how a small model like this does as an alternative to Claude/GPT4.
I tried the OCR and OCR with region on images converted (no scanned) from PDF pages. Nothing fancy, standard text with some titles, sections, lists... it is absolutely unusable. When it detects something, it usually got it right, but it could only see around 25% of the text.
Hi Sam. Thank you for the videos. I've been playing around with some of the smaller vision models and trying to implement batched inferencing with little success. If you were trying to accomplish running multiple VQA style questions against the same image quickly, how would you go about that goal? Is batching even in the right direction I should be looking?
I wonder how much performance would be affected when something so distilled then gets quantized? Also, it seems amazing that it can handle segmentation for an unspecified set size! With Phi3 Vision you would need to provide a token to represent, say, each giraffe you want to identify.
Hi Sam, thanks for the video. What do you think about how does it compare with Phi3-V? My take is that this is more raw and better for fine tuning, do you also think so?
I'd love seeing a fine tuning video, specially if it's not question answering, just so it's a different use case from the documentation. Maybe with a quick intro talking about what are possible scenarios where fine tune would be specially helpful.
This is what people should call "small", anything below 1B! Thanks for your video. By the way, I played around with the quantized version, the result is unbelievably good! I shared a post on Twitter and mentioned you and shared the Colab. Take a look at it. I tried 8 bits and 4 bits. It's odd how 4 bits is almost the same as the base model!
Thanks for the great content. A video going through the fine-tuning process on this one would be amazing. I am not sure how this could scale to a video implementation (probably passing a frame each time).
I also would love a video/notebook for a Florence 2 fine tune
Another video which I appreciate a lot! thank you for sharing.
I would also like it if you could do another going through the fine-tuning process :)
Thanks for your work on sharing this information. Much easier to watch your content than keep my ear to the ground all day trying to keep up. Much appreciated, sir.
It's also good at OCR for hand written documents
Vqa tutorial would be nice!
We request you to do fune tuning on object detection. Because, all llms are useful generating text oupit only. Thanks in advance
Is this really idea to use data created by another model to train your model ? 1:54
Isn't it going to replicate the errors from other models ?
@samwitteveenai please make a fine-tuning video about VLMs such as: Llava, Florence-2 and if possible try to use Ollama so that we can make the inference on local device.
what would you pick for fine-tuning ?
Any specific application ideas?
I've tried this model, describing the image is great. I've also tried the docvqa, but giving only one word answers and not getting even simplest questions right. i had hoped to do some classification and compare with other models.
Is it possible to train ocr task prompt for custom dataset and if we train Florence-2 for ocr task the will it affect performance of the model?
Thanks for the information this is great.
Can i fine tune it for certain specific images like few short learning. Can you put a tutorial for the same it will be great full.
When will you release a demo on to fine-tune such model ?
Thanks Sam!!
Please keep up the great work...
Would be interested on how much memory is required to run these models. they seem pretty small even unquantized. Maybe I will try it later on my 8GB M1 Mini. One thing I am curious about: at 3:38 , the description for the image is wrong in ways that seem odd. The title is described as being on top with the "20 Years of ..." underneath and Ron's tie is described as red and hair blonde. I wonder if this is just vagaries of the model (placement data would be strange) or over reliance on training data. Or a straight up mistake in 'creating' the paper (which would probably be the most disturbing😉).
I think fine-tuning for OCR would be a good demo. OCR in the real world with images of documents is much harder than OCR on electronic documents so would be cool to see how a small model like this does as an alternative to Claude/GPT4.
I tried the OCR and OCR with region on images converted (no scanned) from PDF pages. Nothing fancy, standard text with some titles, sections, lists... it is absolutely unusable. When it detects something, it usually got it right, but it could only see around 25% of the text.
@MH-ke2wi yeah also been struggling to get decent results with OCR
Thanks a lot for this I wish you could consider the continuing process for identifying authentic and fake certificates 🙏🙏🙏
Thanks, Sam! I always appreciate your videos.
I would love your take on how Florence-2 compare with Apple's 4M-21.
Hi Sam. Thank you for the videos. I've been playing around with some of the smaller vision models and trying to implement batched inferencing with little success. If you were trying to accomplish running multiple VQA style questions against the same image quickly, how would you go about that goal? Is batching even in the right direction I should be looking?
It would be great if you can show a finetuning example!
I'm enthusiastic about these smaller models. Thanks for covering this!
Please do fine-tuning for Object detection
Great, yes, fine tune would be very interesting.
I wonder how much performance would be affected when something so distilled then gets quantized?
Also, it seems amazing that it can handle segmentation for an unspecified set size! With Phi3 Vision you would need to provide a token to represent, say, each giraffe you want to identify.
quantization is a good question! I would expect it to suffer more than a big model. Might give it a test tomorrow.
Where is the dataset? I couldn't find the release
Thank you - it looks interesting:)
Hi Sam, thanks for the video. What do you think about how does it compare with Phi3-V? My take is that this is more raw and better for fine tuning, do you also think so?
this is completely better and more advanced than phi 3 v crap image detection
Thank you so much!!
I'd love seeing a fine tuning video, specially if it's not question answering, just so it's a different use case from the documentation. Maybe with a quick intro talking about what are possible scenarios where fine tune would be specially helpful.
Noted!
Yes, I'm trying to use it for table extraction out of scanned pdfs with little success so far. Would love to see how you implement that.
This is what people should call "small", anything below 1B! Thanks for your video. By the way, I played around with the quantized version, the result is unbelievably good! I shared a post on Twitter and mentioned you and shared the Colab. Take a look at it. I tried 8 bits and 4 bits. It's odd how 4 bits is almost the same as the base model!
I saw you tweet and retweeted it, very cool stuff. I will check it out. just been knee deep in Gemma stuff for last few days
@@samwitteveenai Thanks, and yes, it's Gemma2's turn. Waiting for your TH-cam notification about the Gemma video!
awesome, thanks