Faster Video Generation in A1111 using LCM LoRA | IP Adapter or Tile + Temporal Net Controlnet
ฝัง
- เผยแพร่เมื่อ 27 ก.ย. 2024
- Computer Specs: RTX 30708GB Laptop GPU, 16GB RAM, nothing else matters.
Contents include:
Sample results
Generation using LCM LoRA + Tile+Temporal+Soft edge controlnet
Generation without controlnet using LCM LoRA
Generation with using LCM LoRA + IP Adapter and other control nets
Notes about Davinci resolve
- In this video, we will be using LCM LoRA in Automatic1111 in order to generate Videos 3 to 5 times Faster using Video to Video method in img2img.
- Img2img video generation is simple, doesn’t require extra extensions to use, and gives us the ultimate control over the generation as I have explained in a previous video. • AI Video 2 Video Anima...
- We will see LCM without Contorlnet, and some nice Control net combinations that may produce some interesting results.
- While LCM Sampler is not yet implemented in Automatic 1111, LCM LoRA still produces good results using Eular a sampler for example as we will see in this video.
🎯 Key Takeaways for quick navigation:
00:00:01 samples videos and intro
The video discusses using LCM in Automatic 1111 to generate videos 3 to 5 times faster, focusing on image-to-image video generation, which is simple and doesn't require extra extensions.
00:01:08 Getting frames using Davinci Resolve
Demonstrates using Davinci Resolve to generate video frames and prepare them for image-to-image generation.
00:02:40 LCM LoRA usage
It shows how to use LCM LoRA for image-to-image generation and adjust parameters like sampling steps, CFG scale, and control nets.
00:04:18 Controlnets (Tile, Temporal,Softedge)
Setting up Tile+ temporal net and soft edge or open pose for enhanced image control
00:06:51 Batch image processing
Cover generating frames, checking their quality, and using Topaz Photo Studio for image adjustments.
00:09:43 Davinci resolve compsition tips
Adjusting video speed, retime, and scaling settings in Davinci Resolve to enhance the final video quality is discussed, show optical flow for frame interpolation
00:14:19 IP Adapter usage
The video mentions using IP Adapter for more style transfer and control in image-to-image generation.
00:17:15 Conclusion on LCM LoRA
LCM LoRA is recommended for faster video and image generation, but it's noted that it may not work well with Anim Diff and requires more experimentation in A1111, and conclusion.
see first video in • AI Video 2 Video Anima...
see LCM LoRA explanation in • Super Fast Image Gener...
see controlnet guide in • Complete Controlnet Gu...
A1111 and stable diffusion beginners guide • Beginners Guide for St...
download LCM LoRAs
SD 1.5 LoRA model
huggingface.co...
SDXL LoRA model
huggingface.co...
download Temporal net Controlnet models
huggingface.co...
download IP Adapter
huggingface.co...
www.blackmagic...
Now unfortunately, LCM didn’t work well with animatediff on Automatic 1111 in my tests, possibly it requires LCM Sampler and some updates, so I recommend sticking with ComfyUI for Animatediff, it also requires better GPU to be used with Controlnets which is important if you plan on controlling your video output.
So LCM LoRAs allows us to generate videos and images faster in stable diffusion, which is an worth trying and using
all videos used in this work are free from freepik and pexels.com, you can download them from
www.pexels.com/
and
www.freepik.com/
thanks to all authors who have created these amazing videos and for their hard work.
www.freepik.co...
www.freepik.co...
www.freepik.co...
🎯 Key Takeaways for quick navigation:
00:00 🎬 *The video discusses using LCM in Automatic 1111 to generate videos 3 to 5 times faster, focusing on image-to-image video generation, which is simple and doesn't require extra extensions.*
01:12 🎨 *The video demonstrates using V Resolve and Photoshop to generate video frames and prepare them for image-to-image generation.*
02:48 🖼️ *It shows how to use LCM Laura for image-to-image generation and adjust parameters like sampling steps, CFG scale, and control nets.*
05:14 🧩 *Setting up temporal net and control net for enhanced image control is explained.*
06:52 ⚙️ *The video covers generating frames, checking their quality, and using Toas Photo Studio for image adjustments.*
09:39 🔄 *Adjusting video speed, retime, and scaling settings in V Resolve to enhance the final video quality is discussed.*
10:49 🔮 *The video mentions using IP Adapter for more style transfer and control in image-to-image generation.*
15:14 🤖 *LCM Laura is recommended for faster video and image generation, but it's noted that it may not work well with Anim Diff and requires experimentation.*
thanks, will try to include these in future videos, will do a reverse operation as well if i got the time later, thank you
This is such an awesome tutorial. Just found your channel. Excited to binge watch all of your videos. Thank you for sharing!
Thank you, hopefully you will find something useful here and some cool learning tips.
This is so good. ai imaging is so fascinating. Thanks for showing us how it works.
You are welcome, yes it's fun and interesting and will get better and faster overtime.
영상을 부드럽게 만들수 있을까 했는데.. 여기에 방법이 있군요.. 당신은 최고입니다
Another great video from you! Thanks a lot for sharing this, great in-depth info!
Thanks for you kind remarks, hopefully it is useful for some.
Hi.If you don't want to wait for CN to be loaded and unloaded in Automattic, you can go to the settings and set the slider for CN-cache (I don't remember exactly), then you will have CN in memory all the time.But it takes more memory, but generation is faster. Also Optical Flow is in Deforum.And you will need to insert the input video into CN and into the "init" tab. Temporalnet 2 has also appeared. But in order to use it you need to configure something in Automatic.
Have a nice day
thanks for the info, i dont think it works with 8GB VRAM unfortunately, indeed load and unload make things take a long time, temporalnet 2 file size is also very huge 5.7GB, that could be an issue on my Laptop as well... hopefully soon we get more optimized networks, otherwise, i should start using Runpod more often :)
super cool bro ! Thanks a lot !
please do one for comfyUI🙏
Wow! Incredible tutorial! So much care and precision. I’m sure this video took a while to make + running your experiments. Thank you!!
(Btw, how much VRAM do you have?)
Thanks, true, 8GB VRAM RTX 3070 Laptop GPU.
Latest Animate Diff update added the LCM sampler.
thanks for the info, will recheck that on A1111, my recent tests on A1111 were not nice, will try again with a fresh install
Nice tutorial bro!
tried with sd forge, it works perfectly..thanks man.. as you all have python installed you can use the module "moviepy" to get the frames of your videos but also generate the video with the generated images after
edit:
i wonder if there is a way to use it in txt2img so we can use openpose rather than softedge so we can have more freedom on what we want (like the environment)
You are welcome, adnimate diff works better in Text2img with open pose for example, it gives more freedom over the environment, but requires more computing power, better GPU.
it's possible to have LCM A1111 , by adding some line codes in two files of a1111
Yes, i did that, I saw a post somewhere and followed it a while back, didnt see results to be an improvement over euler a
sir plzz tell me how create videos like bryguy
Good video. I was just wondering what it would be like to make animations with LCM lora. Do you know how an animation could be made with a specific face while preserving its hair, beard, eyebrow, lips, nose... would I have to make a lora (like you have in other video with Elon) or could I do it with an image?
Yes, possible, Currently IP Adapter controlnet allows you to morph a face check th-cam.com/video/k4ZWJD6W8d0/w-d-xo.html where i explain an example with IP Adapter, you just choose the model to be IP Adapter Face and put the face instead of the full body in the first controlnet .... or use face swap technology such as ReActor as in th-cam.com/video/gwId5NUNKDk/w-d-xo.html ... making a LoRA for a person really takes time and lots of experimentation, still, best results are achieved using a LoRA with After detailer (but it can takes days and lots of trials to achieve perfect LoRA for a person)
use reactor face swap, formerly ROOP.
I am trying to create LoRA for characters and clothes seperately. I have seen both videos of LoRA clothes and character. Is there any sure shot settings to create LoRA for characters which gives best accuracy in the result image? Because I need to automate the LoRA character where I will just need to select 5 6 images of the person and rest process can be automated?
Same goes with clothes training LoRA? Can you suggest something to do so? Is it possible? I am training LoRA to get most realistic and accurate face but some faceswap results are better than generated images. Any Suggestions?
IP Adapter Controlnet, which allows face swap, style application, you might want to google that out
unfortunately, based on what i have seen, LoRA training doesnt always produce great results, requires testing different settings in some occasions, but it yields better results when done right than face swap... I dont know of any tool for automating the process either...LoRA in general, might take time, because same settings may not work for different datasets, even results produced by some checkpoint might be better than another checkpoint, so lots of testings is required to produce something really good with LoRA.
using IP adapter controlnet inpaint, right? but this is the manual process to mask out face and dresses. I think i will need to find the face edge detection model and parse the image to it, and then the masked image can go through image to image. That's how i can automate the process. Let me know if you have any approach.
can still tell its ai. if they can get rid of the flickr and changes that would go so well but progress for sure! in a year or two yeahh lol
true, it will take few years even based on current progress before flickering disappears, but i think ,that future Videos will be 3D generated and animated for perfect consistency and zero flickering, because Stable difficution will always produce some flickering, even in more complicated animation methods using Animate Diff and other tools in ComfyUI.
@AI-HowTo Hey any chance people can contact you for proper business discussion? :)
sorry, I cannot at the time being.
@@AI-HowTo Thx for your answer!
Friend, you can do it with XL since it is very difficult to guide yourself if you use SD 1.5, basically it is doing everything differently from your video and I find many errors and deformed images
true, XL is certainly better, but I still dont use it unfortunately on my 8GB video card.
wow looks like what i need thank you
How could I miss this gem of a video for so long. Thank you so much for this mate💛🤝😍
Glad you find it useful, you are welcome.
You haven't mention about the model. Also what to put in the VAE folder??
The video just focuses on how to to things, the model doesnt matter, any model can be used, some models dont require a VAE, so we usually keep the VAE to automatic or select a specific VAE depending on the model specs which tells us if we better use a VAE or if the VAE is baked into the model already... in this video i used a normal model which is aniverse v1.5 ... currently LCM sampler is also official supported in A1111, and there are LCM Models too that dont need a LoRA to be used.
By the way from my own it kept iterating on the same picture for the whole set of frames was in the resolve outputs .... any idea wher it come from ?
not sure, double check that you are using batch folder properly.
@@AI-HowTo thanks for your answer ,
Well i did check as seperated the input folder and output folder, i'll give it a new shot on less frames coz it took some while to process, comparing to your workflow i added adetailer, do you think it could come from there ?
Hi there & thanks for the tut. Quick question. Why does the output image looks better on comfyUi?
there you can use LCM sampler which gives slightly better image than Euler a, in A1111, there are some LCM Sampler implementations, but still not part of the official release of A1111.
I'm not good with logic and prompts, but can you explained this exactly a1111 method on comfyui? thank you
Will do so if i could in the future.
@@AI-HowTo Thank You so much
要是用lcm生成lora会怎样😏
not sure, I tested with other LoRA models and it works well... LCM LoRA is Trained using their own training script, so i guess, if we train using their script we just get a LoRA that can help generate images faster and generate a subject at the same time, i think, i have not tried it.