Animations with IPAdapter and ComfyUI
ฝัง
- เผยแพร่เมื่อ 9 ก.ค. 2024
- I just updated the IPAdapter extension for ComfyUI with all features to make better animations! Let's have a look!
OpenArt Contest: contest.openart.ai/
IPAdapter Extension: github.com/cubiq/ComfyUI_IPAd...
AnimateDiff Extension: github.com/Kosinkadink/ComfyU...
00:00 Intro
00:09 Animated Masks
04:21 Logo Animation
08:48 32-Frames
11:39 Animated Reference Images
15:19 Contest and Conclusion
** Workflows **
Download the masks from: cdn.matt3o.com/uploads/files/...
Logo Animation: pastebin.com/9GbLTxVs
4 Reference Images Animation: pastebin.com/JCcbciyk
Dragon Lady: pastebin.com/ncFybRfE
** Background music **
- "Part A" by Alexander Nakarada (www.serpentsoundstudios.com) Licensed under Creative Commons BY Attribution 4.0 License
- "Manace" Synthwave by Karl Casey @ White Bat Audio (whitebataudio.com/)
- "Last Stop" Synthwave by Karl Casey @ White Bat Audio
Praticamente mi devo studiare tutti i tuoi video step by step perché sono una risorsa immensa per me che ho iniziato da pochissimo a esplorare questo mondo. Il tuo modo di spiegare è meraviglioso, grazie mille.
Dude, your channel is absolutely insane. It has the most advanced tips on ComfyUI right now. Your youtube channel is starting to grow becuase people are noticing you are the real deal. Please keep making amazing tutorials.
The legend himself!
every update you make is awesome!
Awesome stuff Mateo! thanks so much for sharing these workflow ideas, will definitely be playing around with these, and I can't thank you enough for your continued development on IP-adapter!
Thank you for your awesome contribution to the community! Great work!
Well done! I always enjoy your tutorials.
Fantastic work 👍❤️
Great as usual! Thank you so much for your work! 🥇
Thank you for all your hard work on this node!!!!
I really love this focus on generating what you actually want through various tools. Very refreshing compared to the more "black box" approach you often see with SD, and you can see from video to video how learning one thing can be repurposed for different use cases. Wonderful!
Great video, also wanted to thank you for all your work.
Man, thank you for the work you do! No words for my appreciation. You're amazing 👏
So so neat, thankyou for all your work on this.
Thank you so much for the detailed video. It was very nice to see the intermediate steps as well. Love the blinking girl animation 😍
You’re the developer!!! i’ve been playing with it for days it’s absolutely amazing
This video is GOLD!!
Thank you so much for sharing all of your work.
IPadapter became a must have to me.
Thank you so much for the sharing the workflow.
Many thanks, I only just moved from Auto to ComfyUI and assumed it would be a while before I could do much, but your plugins and Tutorials have got me there much quicker.
I used your Logo Animation to drive two random pics I had lying round and love the result. Just the beginning... all the best. and agree with below, Great Channel.
Thanks in my deep heart, I love your videos so much, really appreciate your amazing work,please keep updating！
stoked to see you connected to the OpenArt contest!!
These tuts are the best!
👍
this video introduced lots of new things to learn. Keep up the good work!
this is honestly one of the best explained ai videos ive seen. i'll be following you from now on!! :)
maraming salamat, Mateo! excellent work.
Thanks for sharing, your videos have always been so insightful, you are a true master and sharer!
Awesome stuff! Best nodes out there 😉
Researching for your next video? lol
@@JustFeral 😆
Great work, another very instructive video.
Absolutely amazing !!
love it dude keep it up
Wow! Great work Matt30
This is OpenSource After Effects y'all! CREATIVE FREEDOM FOR ALL!!!! 😛
This is amazing, thank you thank you thank you so much!
Remarkable ✨ - and great to see that you have joined the jury.
'add a bit-a noise, that always helps!' you're the man!
😁
Absolutely nuts, indistinguishable from magic as goes the famous quote
wow, super cool stuff!
Matteo, you've saved my life more times than I can count. Cheers
Matteo Maestro Latente!!! Absolutely amazing!! ❤️🇲🇽❤️
Amazing!
Wow. Another powerful technique.
bro you deserve more subscribes.
Man, there are so many channels about stable diffusion and AI on TH-cam but I'm learning from yours more than all of them combined. Thank you ❤
same here
this is the real stuff. great video
thank you very much~!
What a time to be alive!
very clever
real cool!
damn.. this is a high step up.... i looking for something like this but with real faces
so great
Genius!
simply amazing, i have learnt way more from this 16 minutes video than other videos on similar length. that logo animation simply blew my mind. kudos to you!
My hero
Even more stuff for me to learn lol. Thanks so much for your wonderful work. I understand virtually everything in the video apart from ScaledSoftControlnetWeigts. What does it do exactly?
it removes control to the controlnet and gives it back to the text prompt. Experiment with a few values. You loose a little fidelity but sometimes helps with the animation
Wow, I am just blown away by how useful this has been. To be able to hear you explain the idea behind each step you take is invaluable for learning how to really understand and use these tools. That is something that is hard to find with other youtubers, who are quick to just say "do what I do exactly" without really teaching you the value of each component and node. Great video!
The very last thing is insane, thanks for the workflow! Any hope for an upload of the two images, i.e. woman and dragon, too? :)
you can find them here: imgur.com/a/Rex4tzX
@@latentvision thanks soo much! Have a great day. You deserve so many more subs. They will come, just keep going!!
Mateo- nice work! Where is your Patreon so we can join! FYI- the most popular use case here is animation studios like ours- we want to take existing video and use IPAdpater to add a style and ControlNet and Animateddiff to get precise control over the animation! Keep rocking - Cheers!
Brilliant Matteo! Your videos are masterful! There are no limits to your creativity and wisdom. The resolution of the 16 images black & white for mask is 512 x 512? Did you create them yourself or are they purchased from a design collection? Best regards from Spain. Mario
I've put a link in the description with a few masks
Thank you very much! I see that they are 512x512 like the project. And if the project is 1024x1024 and the images are 512x512, how can you adapt their size, would it be with the GetImage width and height node?@@latentvision
Where did you download IPAdapter_image_encoder_sd15.safetensors? Can't find it anywhere
just download the image encoder linked in the repository and rename it.
hello, amazing video, may you show some tips an tricks to make consistent anime character's face, body and with other controlnet models?
yes, that is in the pipeline
@@latentvision if possible may you make a tutorial on that?
Amazing Tutorial!! A small question: What does the Scaled Soft Controlnet Weights does exactly?
scales the weights over time instead of applying them linearly
Geez... now coffee please!
Is there a ComfyUI node that allows me to render an Output video of the Denoising stage, you know the Preview you see on the KSamplers at the Denoising stage if you have Preview method set to Auto or TAESD is there a way to output that as a video?
Holy shit, this is motion graphics in a few clicks.
Amazing stuff. Can I also use it to change the style of my input video? I mean input a video and an image for how I want it to look?
you can use multiple IPAdapters, one for the "animation" and one for the style. You probably need to play a little with timestepping to get exactly what you need, but it should work
hi! Thank you so much for the tutorial! I tried to apply this method to my vid to vid setup, using the depth mask from the video as masks for IPadapter, but I get this error at the Ksampler stage :
The size of tensor a (135) must match the size of tensor b (128) at non-singleton dimension 1.
it's rounding error, use image resolutions divisible by 8
Hey Matt3o, I've been using batch prompting with animatediff for a while now and I swear I thought I saw that there's now a way to use ipadapter plus to trigger an image at particular frames. I can't find it anywhere now that I'm looking for it though. Is there a way to create an animation with batch prompting and use one image at a time to influence it?
unfold batch is what you are looking for
Hi Matteo, incredible as always! Quick Question, I see you connected the AnimDiff to the checkpoint and out of the AnimDiff to the IpA, then out teh IPA to the Ksampler. Many Workflows I have been working with have the IP connected following the Checkpoint and Loras, then out the IPA to the AnimDiff and then out to the Sample. Are there any benefits to order you have there?
ideally: Checkpoint > Lora > AnimateDIff > IPAdapter > other stuff like FreeU, rescaleCFG
@@latentvision thank you! And what does the rescale cfg node do? Looked really cool in your example.
@@dkamhaji it's a trick to use lower/higher CFG in the Ksampler without sacrificing quality. It often helps with animations and IPAdapter
Hi Mateo, I am studying all of your videos but found When I use 2 pictures and 16 frame masks with Ipadapter during animation process, I always get a colorful mosaic result, even though I have all the models selected and the settings configured the same as yours, how strange this is.And do you have any clue about this? Thanks.
13:20 Hey matteo could you explain why you put '2' in the 2nd Repeat Image Batch? You mentioned 16 framed animation. But you put 6 on the first and 2 on the 2nd.
This is absolutely incredible. I was very interested by the cat to dog transition workflow, but I couldn’t find it. I rebuilt it from the video. Still, it makes an animation with both start and ending image reinterpreted. I can't figure out how to get a final animation with start and end as close as possible as the two source images. Have an idea or some tips where to tweak ? Thanks for any help and for everything anyway !
if you need to be very close you could use tile controlnet, it's basically the same workflow at the logo transition
@@latentvision Thanks! I was stuck on the cat/dog workflow, trying to tweak it. I'm going to check the logo transition workflow then. :)
Hi, can you please explain when should I enable Unfold batch? I have Apply Ip Adapter node connected to Load Image Batch From Dir (Inspire) which feeds it a batch of images. However when I enable unfold batch I'm getting strange restuls that have nothing to do with the input images (the input images flashes as is at the start of the video, then strangeness). When it's disabled it's closer to the input images. So when should I use it? What does it actually do? Thanks!
mostly animations
I loaded your process for the logo, I get this error, when I remove the mask everything works:
Error occurred when executing KSampler:
Input and output must have the same number of spatial dimensions, but got input with spatial dimensions of [16, 512, 512] and output size of (64, 64). Please provide input tensor in (N, C, d1, d2, ...,dK) format and output size in (o1, o2, ...,oK) format.
you probably have an outdated version of either the extension of comfyui
Thanks for the video! Does the IP-Adapter custom node of ComfyUI download all the models or do I have to manually download them and place them in the folder?
you have to download them manually
Hi! I have a question. Is it possible to use the "CLIP-ViT-bigG-14-laion2B-39B-b160k" model with sdxl? I always get an Error of Size mismatch. If it is possible, what do II Need to do to make it Work?
Only one of the SDXL models is compatible with bigG (the only one without vit-h in the title)
Love your videos! I am running on a Mac which uses Metal Performance Shaders and not CUDA or CPU. Therefore unfortunately I cannot use IPAdapter as it uses autocast. Any suggested fixes or ways to run? Thanks
it should work now on a mac, autocast will be on the cpu.
hello! I'm always watching and learning. thank you I have one question,
3:01 This is a question for this section.
I don't see the mask preview+ list. Can you tell me how to install it? I also updated to the latest version.
it's in the ComfyUI_Essentials extension
@@latentvision
Oh my god!! It's solved. thank you!!!!
I'm having so much trouble trying to adapt this to work for SDXL. Could you do a video specifically about your last couple of setups but adapted for SDXL?
AimateDiff SDXL models are not great yet, but I'll make a more indepth video about animations in the future
Hey bud, how can I contact you? I love your work and would like to collaborate on my education company :) Thanks!
matt3o on discord, but I'm well over capacity at the moment :(
This is great - much appreciated. Having a bit of trouble finding the Masks folders for VHS (e.g. zoom 16) ... do you know where to download those in particular?
those are not included, sorry, I still need to set up a repository where you could download all that stuff. But it's very easy to do with a video editing software like KDEnlive. I'll put a link in the description or in a pinned comment when I find a solution
Where do I put the files? I have the zipped masks files but where to put them so the node can find them? @@latentvision
A video from ipadapter sounds great! But the restriction to 244x244 pixel input images are a bit annoying if you're trying to get really specific results, isn't it?
we can only hope in higher resolution image encoders unfortunately
Do you have a channel where we can ask questions? I have been stuffing my brain with as much ComfyUI that I can and I feel as if I may be asking a silly question here... how do I get an image as you did where everything in the image is the same except her eyes being opened or closed?
I'm thinking of opening a Discord but I'm well over capacity and I'm a bit worried that it would suck too much of my time... I need to think a bit about it
@@latentvision I think your discord is going quite well!! So glad you took the leap! Thank you!
Been trying to follow along with your last animated example. but with SDXL i've not been having too much luck. Could be a great video if you show a few of your things for the SDXL workspace
SDXL AnimateDiff models are not great unfortunately, but yeah I gonna talk about that too
what is difference from running model through ipadapter before animatediff vs after?
ideally it's: checkpoint > loras > AD > IPA. AD makes some changes to the model pipelines that are used later by IPA. So just to be sure I'd put IPA after AD
I tried updating ComfyUI and all the custom nodes, but then I'm getting this error AttributeError: 'NoneType' object has no attribute 'to' on the ControlNet.py . Do you have any idea if I have to disable any custom nodes, or install some other nodes?
maybe you are using the wrong controlnet model, there are a couple around
@@latentvision !!!! thanks for the advice, i was using ControlNet LORA. problem solved after switching to the standard controlnet .pth file
I'm trying to figure out how to do this with a video and an image, as in, take a photo of a teddy bear and have the video of two people morph into teddy bears.
1:32 Rude, you sound like my doctor.
I really don´t see why you repeat the image batches 2x at 13:39, could you please explain? thx.
yeah it's not really needed, you can reuse the same batch twice
Hello, why don't I have your mask selection in my image loading node?
I'll post a link to the masks. Sorry if I didn't before
@@latentvision Thank you very much, I think it is great, I like your tutorial very much, I learned a lot from it ❤
Hi i cant see what CLIP vision model is loaded @ 1:16
the sd1.5 one
在2：11，VHS_loadimages的directory选项中怎样做才能出现您在MASK文件夹中的目录，它们需要放在什么地方？希望等到您的回复。
copy the video (or the frames) in the comfyui "input" directory, or alternatively you can use the "load video path" or "load images path" node
谢谢@@latentvision
Ahhhh... thank you... this was what I was looking for. @@latentvision
Any repositories you know of for various sets of masks?
check the video description :)
Hi Mateo,
Can the IPAdapter weight be scheduled?
not by a Jedi 😄
no, not at the moment, but I'll make it possible in a future update
Maybe I'm stupid, but you can basically make an entire movie with this, using caracter morphs, movements, impaiting...
I can not use this efectivly because I have a cheap pc, but I'm about to pay some money to have a dedicated machine and build a workflow.
I just have 1 question, imagine you have that last schema, you save it and load it whenever you want and make the same kind of generation, that part I understand, but is there a way to for example get the output of that schema with the girl blinking her eyes, and continue in another workflow or schema without the need to manualy save and load the other schema? Can you queue it and then make another flow, queue it to but the second queue generation depends on the first one? Can you do that with ConfyUI? If so can anyone point me for any video or reference on how to do that?
Thank you for your content, for your work with the IPnet, and for any response in advance ;)
you can link as many nodes as you want in a workflow... as long as you have enough system resources you can link any output image to any new image generation
I was thinking more at using the workflows as separate functions
Like a way to connect a workflow to another workflow that is not loaded, but the way you said that gave me another view on this, just make all the workflows inside of one, and activate it when need.
But yeah I think it will be very memory intenssive keeping a huge workflow with lots of nodes
Thank you for your feedback ;) @@latentvision
@Latent Vision pastebin links are broken.
Grande Matteo , io uso AnimateDiff su SD 1.5 secondo me le animazioni sono ancora ottime, ma tu puoi farmi un corso per ComfyUI (pagando) ?
avessi il tempo... volentieri...
Is it possible to run these workflows on an 8gb VRAM card?
if you are very careful with your resources, yeah. even better on linux
@@latentvision Thank you. Do you have a video on your channel covering that?
What is the channel of DISCORD?
I'm thinking of opening a Discord but I'm afraid it would suck up all my very little spare time I have left... I'll let you guys know if I find the courage to open one
Yeah, it is cool and all that, but it is still working with words and not shapes. And artists are working primarily with shapes, and this is what industry is requiring them to do. I would jump into this as soon as it will be more close to a 3d modelling or sculpting :P Think of neural networks more as a very densely archived asset library of materials, noise patterns, photo realistic surfaces, etc. But I don't like when you can assign any word to some series of shapes or images. Not like I am going to whine about ethics and where these "shapes" are coming from once again, but think about censorship for example, and how people who trained the particular dataset can limit your creativity and control your canvas through the words.
for shapes you have controlnets