This isn't really accurate. LoRA doesn't really make your model smaller, it makes the fine tuning process require fewer resources. As far as I understand the final model size is about the same, if not slightly bigger.
A better analogy I assume with the lego thing, would be that, it gives you a little handbook on how to build that specific thing with the legos. Not handpicking certain legos.
Final model is slightly bigger as it has the LoRA layers added on top of the original layers unless you removed a few of the original before adding the LoRA ones. The training process however requires far less computational resources cause you're only training the newly added LoRA layers and freezing all the other model layers.
This makes no sense. Keeps saying models over and over with no info on what it h means. I get the overall point of being limited. But no info on what it actually is it what it does.. Essentially pretty much know nothing about what lora means
Only watched the first lego example, and that is just plain wrong. LoRA does not make the model smaller, that analogy does not hold at all. Don't want to know what follows from such a bad analgoy.
Training only - you may encounter latency issues during inference due to separately loading the base model and the LoRA model, but most libraries will allow you to merge them, so it would end up being equivalent to the original model.
To actually run inference and generate predictions after fine-tuning Lora, you would need to combine the original large base model weight matrices with the small updated factor matrices that Lora learns. So at inference time, you still have essentially the same enormous number of parameters as the original foundation model. The key efficiency gains are seen during the adaptation/fine-tuning process. By only updating a tiny fraction of parameters, Lora allows much quicker and cheaper adaptation compared to full fine-tuning. But once the adapted model is ready for deployment and inference, multiplying/merging those factor matrices back into the original base weights results in effectively the full set of parameters at inference time.
really good video for a high-level understanding of the concept. I wish there was a little bit of math included as well, just very high level mentioning what mathematical steps are taken.
I’m trying to follow your recommendation of learning python, I’m not a programmer whatsoever, so just from watching this overview, which is great by the way. Is there LoRA-esque way of learning python , so that I can be creative ,as I learn python. Just like using LoRA’s in SDXL, in combinations to create new image recipes. I hope to learn python so that I can get involved with making LoRA’s for sound design, not grounding breaking, but definitely enabling sound designers to create with new sonic colours. Any advice would be much appreciated. Thanks in advance Wes / comments community.
This isn't really accurate. LoRA doesn't really make your model smaller, it makes the fine tuning process require fewer resources. As far as I understand the final model size is about the same, if not slightly bigger.
he meant during training I guess, but you are right, Lora is for fine-tuning
A better analogy I assume with the lego thing, would be that, it gives you a little handbook on how to build that specific thing with the legos. Not handpicking certain legos.
Final model is slightly bigger as it has the LoRA layers added on top of the original layers unless you removed a few of the original before adding the LoRA ones. The training process however requires far less computational resources cause you're only training the newly added LoRA layers and freezing all the other model layers.
This makes no sense. Keeps saying models over and over with no info on what it h means. I get the overall point of being limited. But no info on what it actually is it what it does.. Essentially pretty much know nothing about what lora means
Fine tunning?
Stacking and transfering?
Thanks for a great video.
Astonishing content Man 🚀
Only watched the first lego example, and that is just plain wrong. LoRA does not make the model smaller, that analogy does not hold at all. Don't want to know what follows from such a bad analgoy.
That was excellent. Nice teaching skills, Wes.
Thank you for this clear explanation!
You explanation is really amazing one can easily understand without having any background knowledge, thank you very much.
Is there a speed improvement during training only or also at inference?
Training only - you may encounter latency issues during inference due to separately loading the base model and the LoRA model, but most libraries will allow you to merge them, so it would end up being equivalent to the original model.
To actually run inference and generate predictions after fine-tuning Lora, you would need to combine the original large base model weight matrices with the small updated factor matrices that Lora learns. So at inference time, you still have essentially the same enormous number of parameters as the original foundation model.
The key efficiency gains are seen during the adaptation/fine-tuning process. By only updating a tiny fraction of parameters, Lora allows much quicker and cheaper adaptation compared to full fine-tuning.
But once the adapted model is ready for deployment and inference, multiplying/merging those factor matrices back into the original base weights results in effectively the full set of parameters at inference time.
This video needs more love. Thank you man, so well communicated.
i subscribed to your channel after watching your explanation. Thank you for your clarity.
What kind of tool is used making the animation?
Is it possible/reasonable to use LoRA w/ GPT-4?
Afaik GPT-4 is closed-source
how is Lora fine-tuning track changes from creating two decomposition matrix? How the ΔW is determined?
all that images-filling by scribbling inside the lines? How does that support your content (which in fine)?
really good video for a high-level understanding of the concept. I wish there was a little bit of math included as well, just very high level mentioning what mathematical steps are taken.
1. a large beer cask. 2. a measure of capacity, usually equal to 252 wine gallons. Otherwise well done!
Hello, what is the name of the software you use to draw?
+1
nice explainer
I’m trying to follow your recommendation of learning python, I’m not a programmer whatsoever, so just from watching this overview, which is great by the way. Is there LoRA-esque way of learning python , so that I can be creative ,as I learn python. Just like using LoRA’s in SDXL, in combinations to create new image recipes. I hope to learn python so that I can get involved with making LoRA’s for sound design, not grounding breaking, but definitely enabling sound designers to create with new sonic colours. Any advice would be much appreciated. Thanks in advance Wes / comments community.
518 Skiles Shores
Satterfield Hill
204 Oberbrunner Valleys
458 McGlynn Circle
50849 Klocko Plaza
Hal Crescent
290 Block Place
Pieces of Lego are not called Legos
Elisabeth Field
*powerful not powerfull :)
Unliimmmmmited powaaaaaafulllllll 😁
Jerome Groves
useless video, you can make like ~10 videos by just replacing LORA with anyother AI jargons
its not true
just wasted 4 minutes on this