Using GPT-4o to train a 2,000,000x smaller model (that runs directly on device)
ฝัง
- เผยแพร่เมื่อ 21 ก.ย. 2024
- More here: www.edgeimpuls...
The latest generation LLMs are absolutely astonishing - thanks to their multi-modal capabilities you can ask questions in natural language about stuff you can see or hear in the real world ("is there a person without a hard hat standing close to a machine?") and get relatively fast and reliable answers. But these large LLMs have downsides; they're absolutely huge, so you need to run them in the cloud, adding high latency (often seconds per inference), high cost (think about the tokens you'll burn when running inference 24/7), and high power (need a constant network connection).
In this video we're distilling knowledge from a large multimodal LLM (GPT-4o) and putting it in a tiny model, which we can run directly on device; for ultra-low latency, and without the need for a network connection, scaling to even microcontrollers with kilobytes of RAM if needed. Training was done fully unsupervised, all labels were set by GPT-4o, including deciding when to throw out data, then trained onto a transfer learning model w/ default settings.
One of the models we train has 800K parameters (an NVIDIA TAO model with MobileNet backend), a cool 2,200,000x fewer parameters than GPT-4o :-) with similar accuracy on this very narrow and specific task.
The GPT-4o labeling block and TAO transfer learning models are available for any enterprise customers in Edge Impulse. There's a 2-week free trial available, sign up at edgeimpulse.com !
![The moment we stopped understanding AI [AlexNet]](http://i.ytimg.com/vi/UZDiGooFs54/mqdefault.jpg)
Commercial use here would be a cat door for cats to get in/out of the house, and check if the cat has an object it its mouth, like a dead mouse, rabbit, bird, neighbor kid, etc, then not allow the door to open. Otherwise allow cat in.
That's a great idea.
Data for a dead neighbour's kid might be tough to collect.
Lol! Excellent idea, but I wouldn't be surprised if in a few months the cats are using AI to keep us out of our own homes.
Fun Fact :- No one cares about cats....
Really exciting concept. Watching that had just stretched my mind with the intersection between cloud and device.
Dear Jan, I just found your video in my feed. Recognized in the first minutes how much quality your content has. I took a look at your company and did not know Edge Impulse before. Even if I am yet a personal user of AI products I really got interested as I am planning to change my career path to the AI branch, as I think this is the right way. So to give you a feedback: If you create videos like this one, where you speak to the audience and show simple use cases as an example, you will attract more people. I will take an in depth look into your company now, it looks very interesting! Salutations and keep pushing!
I've been looking for this for a couple months, now I see this video, my mind is blowing!
Agreed! Humans are never satisfied, isn't it wonderful! :)
Solutions are there, if you look...
This is a fantastic demo, really compelling. Can’t wait to try this out. Subscribed!
This is the way for distribution and actualization of LM impact in more real world scenarios
the more I'm watching AI videos the more.... i encounter extraordinary peoples who are really share their knowledge... thank you.
A real world use case i was thinking of a few weeks ago was a grocery store assistant. likely built into AR glasses, you would first build up grocery list then the model would look for those items via the camera as you're walking down the isles. probably would take a long time to train with all the different items you would need to account for but solves one of the biggest gripes i have with going to the grocery store.
use large and mature modal as a labeller for specialized smaller trainning set.... very interesting thought!
I was expecting a trimmed LLM model running on a SOC, but it is nothing but a labeller.....
and counter to the usage agreement you signed when creating your openai account....
Using video frames for training is really effective for small domains. Not only for Yes/No answers but also generative models. Before the release of GPT3, I was playing with the original Diffusion code (available on github), and giving it frames from a very short dashcam drive-around-town. The results were amazing. And this is for unlabeled data. I used this same data to train a VAE, and that just sucked, definitely some magic going on in the diffusion model, even for very limited data sets.
Super cool application. Very impressive.
You've now seen an example of overfitting en.wikipedia.org/wiki/Overfitting. The results aren't related to GPT at all. You could manually label the images and achieve similar outcomes. While GPT and its API can assist in labeling datasets at minimal cost, don't be misled by what you see. The fine-tuned network likely wouldn't work for new, differently designed toys or in different environments. You would need many more videos for a production-level classification solution and would end up spending more on GPT's API calls.
I'm guessing that's what they're doing - use GPT for labeling and then train a smaller classification model with data labeled by a bigger model.
@@Caellyan What he meant is that the model trained in this video is only capable of recognising the toys it was trained on. It won't recognise any new toys. To do that, you'd need to train the model with many many more examples of toys. You could use GPT to label all those new examples, but would it be worth it?
"Overfitting" depends on what you want to fit. If you are making a robot that will only move around in that house, and will only see the objects in that house, then this works perfectly fine, no overfitting. There are many industrial usecases that are equally specific as this. If you claim that this is a universal toy detector, then you could say it is an example of overfitting. But this is a "toy in this persons house detector", and the tests show it generalizes beyond the training data (but still in this persons house).
What'd be more interesting is having the LLM also segment the image automatically, although this can already be achieved through SAM or similar. I guess that it's overall an interesting direction to training small specific models like Yolo 😊
@@petersobolewski1354 Language models don't work with images.
Note that your 'toy' prediction is based on your training data and not on any other content than what it was trained on. Any other toys would never be recognized. Therefore, the model is very limited. However, for your purposes, it is certainly suitable and clearly illustrates how small a model can become when it is restricted to specific purposes.
Is that the case? What's the purpose of the transfer learning step with the MobileNet v2 backbone then?
oh how could they not foresee such a problem and launch the service
That's the point, you're "overfitting" your model so it is very efficient and lightweight on your data, let's say detecting when your car is parked by using your security camera. Perhaps it is not very impressive, but it does its job and only its job. Low latency and low resource with the cost of some accuracy and ability to recognize things it was not trained for. Basically, you're just making a goon that does its job at a minimal cost
Note that no one expects that.
He said factory pretty often for a reason I guess.
S T A N D A R D I Z E D
So that thing has a pretty good chance to work well.
(Imagine the same picture everyday from a camera that just sits in a backyard “did (XXX) change? As a prompt- and footage over some full years as training to not get confused with seasons)
Yes, it is possible to use a multimodal AI assistant that can both see your computer screen and respond to voice commands. This type of technology is often referred to as "vision-enabled voice assistants" or "visual voice assistants."
Some popular examples of such assistants include Amazon Alexa with the Amazon Lookout for Gadgets service, Google Assistant with its Vision API integration, and Microsoft Cortana with the Windows Eye Control feature (which requires specialized eye-tracking hardware). These assistants can perform various tasks such as identifying objects on your screen, providing visual feedback based on voice commands, or even controlling your mouse and keyboard using only your voice.
However, keep in mind that these features may require additional setup and configuration, including enabling accessibility settings, installing necessary software or services, and granting permissions to the AI assistant to access your computer's camera and microphone. Additionally, privacy concerns should be taken into consideration when using vision-enabled voice assistants, as they involve sharing more personal information than traditional text-based voice assistants.😎🤖
Great Demo. Some potential applications would be monitoring turnstiles for employee IDs for access to restricted areas, wearing safety equipment in a warehouse, authorized vehicles in a lot, speed of vehicles/forklifts, listening for unsafe sound levels in an area? all based on visuals or sound. What models did you have in mind demonstrating sound monitoring? Great Job !!
Very cool. Don't be shy, tell us the rough costs of training and downloading. I reckon we might be able to use it to recognise when a Kite is flying the way we want
Really impressed by this feature. I will try it out with my students.
8:50 you could improve the result with
:
1. a third 'background' category,
2. or filter it through an anomaly detection when there are no indicator objects
(perhaps putting a trash hold on the soft-label values)
3. or edit the dataset so that it can see more empty parts where the label is no.
This is a great example of how AGI will almost immediately lead to ASI.
So you basically used the GPT-4o for labelling the training dataset (that you created manually), that you then used for training your small model. 😊
Most importantly he extracted the frames from the video and trained on those. Training on video frames works exceptionally well, whether they're labelled or not. I suspect the reason is because the same object appears in translated positions from frame to frame. This enables the model to discover specialized convolution filters that generalize for detecting that object.
True use case of convergence of Generative AI and Embedded Machine Learning. Very Nice...
You trained a classifier for the combination of your house, your type and style of toys, your camera, and your lighting conditions.
If someone would run that in a very different house (eg one with more patterns and colors) and different toys it would surprise me if you would get great results at that model size.
As I understand that is the purpose of it, to be as specialized as possible. But keep in mind that what he did there was just an example. Obviously you should use way more case scenarios and pictures. The beauty of the system is just really the way the LLM can label the the training data. This is not supposed for deployment so anyone can use it, but for personal use, or internal use in a factory or office. One thing that could be use for would be gun detection in an image, maybe someone getting hurt or in hurms way.
This is so cool, congratulations on that!
maybe someday, an edge device can get an upgrade from cloud LLMs if there's error happens, by uploading the new error video for cloud LLMs to learn & transfer back to the edge. sounds interesting in IoT with giant LLMs network
Amazing Application, I would think about apply it to my screen reading for faster processing.
I already use your platform! Great idea to explore! Thanks for sharing
I paid for Enterprise but the Pipeline isn't working. I've 1178 data images in training, but none of them are being labelled because the Pipeline is registering 0 samples. How do I fix this?
It would be nice if the webapp could take a live video stream from the camera for the inferencing.
This video is an ad.
Why did I get it in my feed not tagged as an advertisement?
Well, the demo is cool, but how well would this work on a new set of images/video. If it's only fitted for this particular video or for images very similar to the frames in the video, I do not see how it could be useful.
It will not work. It was trained on a degenerate dataset and can only interpolate this particular toy on this particular bed in that particular position.
The question is, if the dataset was 10x or 100x larger with more varied data, could it distill a tiny model with more generalized inference? And if that is the case, how expensive will this be to train using GPT-4O?
It will retain some generalised behaviour because we don’t train a model from scratch, but rather use a pretrained frozen backbone (here MobileNet). So rather than just map pixels to “toy or no toy” we force the model to learn that behaviour based on general embeddings trained on a huge dataset - overfits a lot less, and generalises better. However, yes, this model is not going to be great at detecting toys at completely random places - but it often does not matter. Most customers actually have constrained problems (just need to find out when X happens at site Y) and thus constrained models just specialized at detecting this at that specific site are fine. Constrained models for constrained problems. Can always expand this with more training data of course, we’ve had one customer use 5TB of raw data to eventually train a 80kB small model and beat existing state of the art models by 10 percent point (after heaps of signal processing and clinically validated labeling, not with LLMs).
@@janjongboom7561 Exactly what I need. A model that can look at a sound signature and give a yes or no if it fits a certain set of patterns.
Will it run on a Jetson?
it's a great tool to evaluate single states of things on single locations... like is there people on this room, is the water boiling, are the dogs hungry/over the table... whatever single thing you want to check programaticaly
greate idea if we do the same with small robots to help the bigest one that can change the tuff jobs to be more easy.
Thanks for share.
Hi, if you have a very tiny dataset, why don't you use TensorFlow? Newbie here...
Loved it ❤
that opening safety concept won't occur as the standards and regulations are today - there is a thing called Functional Safety that applies, parent standard is IEC61508 and any time a programmable machine might harm a human it should be applied, and it is not quick to apply. Legislation would need to change to do anything else.
This is insane!! 😮
This is amazing.
I appreciate the demo, and I enjoyed it thanks. I guess if you distill this, you're paying ChatGPT (or the model of your choice) to label your data. I think the concern is that to do this for a variety of specific small models, it could cost a lot and Edge Impulse prices will always fluctuate according to OpenAI's pricing (or whoever is running the model). You would probably save some money and lose accuracy on running the LLM locally, can Edge Impulse use locally run models?
And my other perspective to offer is that it'll only be good for labelling common things (things the teaching model was taught on), so some limiting of options there.
I really did enjoy your video a lot though, thanks.
Is the way the GPT interprets an image is by using a CLIP model to generate a caption that describes the image, and then GPT just takes and uses this text?
Wow. As they say, what a time to be alive.
Awesome!
Very cool!
NICE!
I had a different, but somewhat similar idea recently.
Filtered LLMs - filtered to relevant topics for a specific purpose.
The reason would be for memory, oc - can still quantize if needed.
So like, if this LLM is geared for a law firm that does personal injury law, for instance, there's ALL sorts of stuff that can be whacked from the model.
we want medical - related to injuries
We want auto and traffic.
We want insurance.
We want similar cases.
We don't want astronomy, non-related biology (like insects and such), chemistry, music, dance, surfing, etc, etc, etc.
Take the LLM and have something that whacks it down, by stripping and rebuilding.
But then again, this is a bad example, because law firms would pay for backend service... (don't care about local llm) - and they don't need the speed like IOT/edge devices
As the poet said, this is f'n awesome :)
So is there a way to connect the micro-LLM to sensors so that for example I could video a self balancing robot , train the microprocessor and then the microprocessor would control the motorspeed to balance the robot?
This is awesome
Great stuff
this is so awesome ................. wow!!
Hi! I listened to you and understood what you said, but I still have no idea how to start with my Raspberry Pi and reproduce what you did there. I would like to do the same thing, but I don't know how. It would be amazing if I could follow your step-by-step instructions. Cheers!
And that's how you spend millions of dollars to make a live "Is it a hotdog?" app. :)
Hey man, don't laugh, that was the original killer app for dk pic detection lol
They would even augment their data sets with rotations and translations.
Really interesting ; wondering if it could be used for uav navigation (vision based) on commodity hardware
Is it possible to have a collection of small models on device with a "director" that choose the model based on the input?
amazing! thx
This is really nice. Is there an open source version available on github?
@edgeimpulse how feasible is it, using your tool, to create a model sophisticated enough to track facial expressions with a raspberry pi equipped with a micro camera ? Would this likely only work with a specific face it was trained with? Thanks for your thoughts
im woundering if this is missleading, as all the toys looked basically the same, strong prime colors. would it have worked with the teddy bear and what would happen if you pointed the camera at your company logo on the wall..
I really love this concept and def gonna try it but how's the license? like if you wanna train a commercial model then will GPT4o license would allow this? cuz as far as i know its not so it only works for personal stuff, please lmk if otherwise
For labeling images you could probably use a pre-trained Resnet101, which is free.
All good stuff.
Realy great!
Wow, this is mind blowing. Great job!
ok. But what will it answer when it sees toy unseen before? I think, CharGPT will say YES, but your model will say unexpected answers )
Awesome!
can i use your website to train a smaller model for a specific LLM task or is it going to be large no matter what? for example, i am doing classifications for companies , i.e. finance, technology, oil and gas... etc, i usually send the LLM the company profile, and it classifies the company into which category it belongs to.
thank you..
You can use sentence/document embeddings, which are like a 1024 dimensional vector for each example, then train a much smaller neuronal network or other classifier on those embeddings only
Just give GPT4o a big text doc of concatenations and get back a list of labels. Since you only get 80 messages every 3 hours, you simply create 1 big doc to get around this limit.
@@u2b83 I tried it, doesnt work well, too many mistakes
that's awesome!
I would say it's not in the model, it's not in the deployment, it's in the method of construction! Is it possible to iterate that to deployment????
ya. we need more of this. llm is too fat. We need slim version of llm which just enought to do a few selected type of tasks.
like local limited voice recognition on some smaller microcontrollers that would understand other languages besides english?
Impressive ..!!!
amazing!
Do we need to have the same account on edge impulse as well as on chat gpt for labelling.
I have been trying to label data but unable to label.
Can anyone help me.
Awesome
In all fairness have you ever read OpenAI terms of service regarding use of their content or outputs to train other AI models? 😅
Do they forbid it?
Nice video. But since I work in the medical field the data needed to create maybe close to good LLMs is hard to get. But I'm in contact with certain authorities... can you build me an LLM that automatically convinces Squareheads and makes them cooperate with the click of a button? Now that would be great. THX³ in advance.
It seems that one could use this process to train a plant pathology AI. This AI could then run on a device that is in the field somewhere looking at agricultural environments.
What algorithm are you using for displaying the embeddings clustering?
Here using MobileNetV2 pretrained model, take the embeddings at 3 layers before the last layer; then tsne over the embeddings
docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition/data-explorer
Thanks!
So will it be equivalent to gpt2 ...3
عالی بود
👌🏼👌🏼
4:00 As a parent, what are your thoughts on this level of free access? Doesn't it seem to be openly invasive? Where is this visual data processed, how securely is it stored? Is it stored? If so, how are users being compensated and provided the ability to utilize the platform without exposing their lives to the highest bidder? What are the limits of a universal stake company's fiscal responsibility?
It's against openAI policy to use gpt to train other models 😅
could gpt-4o actually repeat your voice in the same exact way?
wow, so we can detect bad product directly on Convair belt
Is Chatgpt 4o LLM or Generative AI?
Sam Altman left the chat
Hey! I'm a staff software engineer (mobile) who wants to get into building these AI pipelines. Any idea how long it would take someone like me to reskill? And what's the best way for me to skill up quickly?
this is kind of a dumb post. i'm kind of the world's old time secret leader in procedural generation, as in through the 90s, 00s and a bit more, my procedural poetry and music generators were much more encompassing than any visible work. so i'm kind of like the world's most experienced procedural media experiencer.
my issue is that my work and related factors drew intense abuse from some of the less visible factors in contemporary culture over a sustained period of time, so "trust issues". i'd have to train my own AGI amusements. this is keen to see. what i'd like to say is this is keen to see, technology applied in some kind of useful way for real lives. not what i expect from society whatsoever.
minicpm can run on mobile head an apk installer
But the data base is way too small, if you show any color object to it , like orange it will be toy ( color x pixel ) . So I would say , it's click bait ;)
External storage is simply added as required
*Have tried having the model that chatgpt labeled the data for label data for another model and then have that label data for another model and so on until....?*
incoming openai lawsuit
It could be used to train a model to detect any sexual activities in your house, or where people are supposed to be doing work 😉 , like you get a message immediately.
small language models retire large language models
Nice but its just knowledge distillation!
“just” 😂
I don't think they understand anyway
You can't use openai model to train other model, they can sue you
😅😅
"cloud on the edge" is pure marketing bullshit.
Cloud is another-mams computer
Unfortunately they don't understand a thing. And they are very gullible to availability bias, even if something contradicts logic.
show the how not the what 🤦♂️👎
15 mins of fluff distilled by chatgpt:
Large language models (LLMs) like ChatGPT-4 can understand and respond to real-world scenarios using multimodal capabilities (text, images, audio), similar to human understanding.
Deploying these powerful but large models on the edge (e.g., in a factory) is challenging due to their size, cost, and latency issues.
A solution is to distill knowledge from large LLMs to train smaller, specialized models that can run locally, reducing latency and cost.
Edge Impulse facilitates creating these smaller models for edge devices, demonstrated with an example project detecting children's toys in a home environment.
Smaller models, once trained with LLM knowledge, can run efficiently on simple hardware like microcontrollers, making AI applications more accessible and practical for various use cases.
Use ChatGPT to train a smaller model answering yes or no to the question, “Is there an adult close to the camera holding a baby incorrectly?” Set it up in your kids’ room. How could anyone be against keeping kids safe?
isnt that illegal
So weird that we create people who are so smart in terms of computers and so dumb in terms of ethics and morality.