AnimateDiff ControlNet Animation v1.0 [ComfyUI]
ฝัง
- เผยแพร่เมื่อ 4 พ.ย. 2023
- Animation Made in ComfyUI using AnimateDiff with only ControlNet Passes.
Main Animation Json Files:
Version v1 - drive.google.com/drive/folder...
New Version v2.1 - / update-v2-1-lcm-95056616
Output Render 1 (SD Mashup) : • Models MashUp AnimateD...
Output Render 2 (EpicRealism) : • Collide - AnimateDiff ...
Output Render 3 (Fashion) : • AnimateDiff Control Ne...
My Discord Server : / discord
-----------------------------------------------------------------
Resources Used :
1) ComfyUI Impact Pack
github.com/ltdrdata/ComfyUI-I...
2) ComfyUI_FizzNodes
github.com/FizzleDorf/ComfyUI...
3) ComfyUI's ControlNet Auxiliary Preprocessors
github.com/Fannovel16/comfyui...
4) ComfyUI-Advanced-ControlNet
github.com/Kosinkadink/ComfyU...
5) ComfyUI Inspire Pack
github.com/ltdrdata/ComfyUI-I...
6) AnimateDiff Evolved
github.com/Kosinkadink/ComfyU...
7) ComfyUI-VideoHelperSuite
github.com/Kosinkadink/ComfyU...
8) Dynamic Thresholding
github.com/mcmonkeyprojects/s...
---Checkpoint---
1) Temporaldiff-v1-animatediff
huggingface.co/CiaraRowles/Te...
Put in : ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\models
-----------------------------------------------------------------------------------------------------
Reference video : • 'Collide' Dance Trend ...
-----------------------------------------------------------------------------------------------------
◉ I also post all my Secrets techniques, tips & tricks, workflows, project files and Tutorials on my Patreon account : / jerrydavos ◉
-------------------------------------------------------------------------------------------------
Music Used :
Vintage Lukrembo - Dream With Tea (freetousee.com)
Stream Universal - Vibe Tracks (Soundcloud)
Dstrion - Alibi (ft. Heleen) [NCS]
Stream skyline - PHONK BEAT (SoundCloud
-----------------------------------------------------------------
SEO:
Animatediff control net
Animatediff animation
Stable Diffusion animation
comfyui animation
animatediff webui
animatediff controlnet
animatediff github
animatediff stable diffusion
Controlnet animation
how to use animatediff
animation with animate diff comfyui
how to animate in comfy ui
animatediff prompt travel
animate diff prompt travel cli
prompt travel stable diffusion
animatediff comfyui video to video
animatediff comfyui google colab
animatediff comfyui tutorial
animatediff comfyui install
animatediff comfyui img2img
animatediff vid2vid comfyui
comfyui-animatediff-evolved
animatediff controlnet animation in comfyui
------------------------------------------------------------------- - บันเทิง
![AnimateDiff ControlNet Animation v2.1 [ComfyUI]](/img/n.gif)
You've inspired me to download the whole lot, god help me downloading it all correctly and getting it working... anyway thank you for the video, maybe if I get there you'll see a creation of mine...
I am Feeling proud that I inspired you
You inspired me too but I hope I can find the software. I dance very well but this effect makes it fabulous!❤
Excellent tutorial, one of the best I've seen on stable diffusion. Very descriptive, easy to follow, and the imagery (color coordinating functions) was very intuitive and really helps to understand the process. Thought I should let you know how much I appreciate the extra effort and generosity you implement in your teaching. Going to go watch your other videos and anticipate more tutorials of this quality from you in the future. Thank you, Love and Respect!
Thank you for this generous reply @lylewells5660, it's been a pleasure to make tutorials like which make you feel satisfied to watch ❤️
This video is so clear and concise. Thank you so much.
I saw your video for the first time today. You're fast and to the point. Very good workflow and video. I hope the updated one works with the new controlnets :D
God, this is really amazing. Thank you so much, i was wanting a real way to learn more about this new AI Tools and this video that you maded is one piece of knowledge, hope to see more people learning from you, and that your channel get a lot of success
Glad you liked it! 😊
omg aDetailer post-processing trick was awesome, thanks a lot ^_^
great workflow:)
Thankyou very much, really is one of the better ComfyUI animation tutorials, will take in the rest too. Again Cheers!!😍😍👍
Glad it was helpful!
@@jerrydavos kind of can't tell you how much.
This is all i need thank you so much hard work!
You're welcome 😊
Great works, bro!
Appreciate it!
Thank you so much for share, excelente work
Best Tutorial! Thanks you so much! Music sooo goood❤
You're welcome!
you are the best! please keep making tutorials like this
thanks a lot
More to come!
Good Work by Jerry Davos AI,
well explained and well structured. Keep doing these videos. Pure gold.
thank you
Fantastic. I appreciate that ❤
You're welcome 😊
Thanks a lot. You answer to a lot of my question. Very good job !!!
Glad to hear that!
Thank you so much!
nice video very detail Thank you
This is a very technical and forward-looking guide
bot?
와 요즘 comfyUI에 빠져있는대 아주 흥미로운 영상입니다 감사해요
감사합니다 !
Thank You!!
Damn man your video is incredible . Hope to see you do more videos. Would love to pick your brain. You need a patreon channel and teach this stuff ! Your channel is going to blow up . The presentation is stunning
Brain: Well, I should thank you! I will enjoy it a lot 😈.
Me : btw I already have a patreon account where I post my breakdowns and tutorials : patreon.com/jerrydavos
and thanks I'll keep finding ways to improve and teach it.
Just saw this message right now. I'm going to join soon! @@jerrydavos
I will NOT be attempting this tutorial, but I enjoyed watching it and understanding what this process is like MUCH better than I did before, so thank you very much. The end results are awesome, and I'm excited to watch how this will evolve over the next few years. I've noticed that the soft-edge images result in a VERY similar result to the original video, since they include everything from clothing style to room dimensions. This is very useful when you WANT that, but I wonder if there will be advancements in this area that allow you a bit more freedom to keep certain elements from the original video while changing others more drastically. Regardless, thank you for sharing your process.
Ya, Using Softedge applies the original elements mostly, If It was purely openpose, you can make a skeleton dance without any elements from original video, but openpose markers jumps here and there in every frame of the video which make the body jitterey so hence it is avoided for now, It will develop in the future.
Your welcome!
This shit will be obsolete in about two months. No need to waste time with all this shit.
@@AraShiNoMiwaKo even if true, it’s fascinating to understand these steps along the journey to better and better ai. A lot of this stuff will get automated, but understanding how it works now can help you understand how to get better and better results later as well, I bet.
@@joelface meh you don't need to know how an engine works to drive a car, or anything basically, this is the same. I'm an artist, I'm not into this nerdy thing. And btw this dude didn't even upload his json workflow, so thanks for nothing, show-off.
very detail, nice
Glad you like it! Thanks
thanks for this video
You are amazing ❤❤❤❤
you are great
10/10, thanks man!
Glad you liked it!
@@jerrydavos very much. Really providing a ton of value. Also, your music taste is very good.
One question, if you're open. Could you address how you got so much consistency? I tried this with a tiktok dance video and noticed that the clothing and hair color in particular shifted a lot. Is the answer to this just to put it in the prompt (for instance anime dancing girl at the end, you put in the prompt "black hair, skirt, etc)?
Thanks again!
@@bobsaget420 Thanks for your Compliment I am very fond of music, and Yes Prompts help only up to a certain extent, After rendering from comfyu, I use Davinci Deflicker Fluro Light or sometimes Topaz Video AI for Deflickering.
Super!
🤠
Thanks!🤩
I don’t understand anything, but it’s very interesting
3d motion capture
😂 yeah
Thank you
You're welcome
Thank you. 감사합니다.
thank u , that great 😍😍😍😍🤩🤩🤩🤩
You're welcome 😊
I actually have been looking easy and fast way to make animation workflow ! Thank you. Only one think I am considering is background. when I use plain background (white or green etc) does not listen to my prompt. How can I improve ?
Use the BG changer v2 file:
Tutorial : www.patreon.com/posts/v3-0-bg-changer-97728634
File:
drive.google.com/drive/folders/1Y7FSlNOy1N2z71r0T91Ty8Yrq5IuwolE
Hey you need more than 8 comfyUI exentisons btw! You're missing like 10 of them. When I loaded the .jsn a bunch of nodes were red! Thankfully the manager extension allowed me to install missing node extensions! (not sure if this is because I used V2 on accident)
this was laid out way more complicated than it needed to be.
Who are the people who are writing this software? It has to be people in the industry who are violating noncompete clauses and NDAs. This isn't the kind of thing you can learn to build in between shifts at the auto plant. Whatever the case is,, as a creator I'm grateful for it. It's opening up some possibilities.
Hi, what checkpoints and models should I use for the (black and white) results in pencil or sketch format?
Nice video. I'll send you over a couple of things I did via discord. I didn't use your exact workflow but a similar one from a couple of different deconstructed ones. Can we go over a post-processing in ComfyUI video doing things like facedetailer and stuff there? I started rendering on Comfy so I geniunely hate using A1111. Going to try the cross-fading of smaller batches like you mentioned now.
Ya, sure would love to see. I am finding ways to accomplish face fix in comfyui so you don't have to go to A1111 and would release in V2 of my ControlNet Animation Json workflow
Thank you, bro! Everything works! just one question, is it possible to somehow eliminate these joints between frames 50-51, 100-101, 150-151 and so on? this is very noticeable during transitions in these places, (I have a weak graphics card, so I only count this range) I made a glitch transition in PremierePro there so that it wouldn't be so noticeable
Yes, it can be possible, just render few extra frames for Cross fading / Overlapping.
For example :
Batch 1: 1-50 frames
Batch 2: 45-100 frames
From frame 45-50 fade in the opacity of Batch 2,
5 frames rendered extra for cross fading so it becomes a smooth unnoticeable transition,
Total Output images 105 ( 100 source frames and 5 overlapping extra frames)
I've used this overlapping frame technique here : th-cam.com/users/shortsErXBU5WD0ZY?feature=share
You can even try 10 frames for even more smoothness.
Hope your query is resolved!
Good luck, Keep creating 😊
cdn.discordapp.com/attachments/1172537908633276487/1172607839307567134/image.png?ex=6560ef26&is=654e7a26&hm=77b612da88e7ccfc119e6f1c12e8f8dfab3148763b4766877919006405511352&
Like this
top
this is advanced! can I download all the links on your descriptions and just following your video?
nice tutorial
Thanks
very nice tut , thanx for the config files . can you only make dancing annime girls? i want dancing skelettons and zombies instead
You can surely use only the openpose controlnet pass and enter the "skeleton prompt" and it should be a skeleton dancing.
And XD, dancing girls are just for testing, main application is beyond just "dancing". Next time will keep in mind.
at 2.33 i thought my computer was having a heart attack you little pranker !
hehe
Do you intend to create more tutorials for this channel? My like and wtachtime 100% 💜💜💜
Yes I will, the Version 2 of this workflow which give more details is in process.
The two hed openposes can be used together to accurately capture movements. This is great, but can it also be used in this way with sd?
it works on Sd
Very informative, thank you.
I have both a question, and a suggestion.
Question: for some reason, backgrounds come out garbled or messed up sometimes. Any idea what could be the solution? I've been learning ComfyUI and to animate for a couple of weeks, and haven't been able to get around this.
Suggestion: Have you tried using Zoe Depth in place of Openpose? Or rather, use both together?
During my many hours of learning and testing, I've found that Zoe Depth has insane detail, even better than OpenPose at times. Including hand/arm positioning.
Thanks again!
In longer batch range above 20-40, the minute details gets blending between the frames hence, blurred or smudged background, Solution is to use shorter batch range and stich them in post, and to have a smoother transition, render some extra frames in every batch and overlap-fadeout them while sequencing.
Some models supports longer batch range and some doesn't, I've noticed realistic and semi realistic models works well on longer batches,
Context Length in the animatediff loader also sometimes cause the issue, try putting values 16,24,32.
One more thing I've observed, Prompts keywords and prompt weights can cause this blurred background, only one keyword can ruin the whole picture, So I test changing the prompt with another synonym , negative prompts and totally avoid prompt weights and it gets fixed.
This technology is new, the best way to overcome bugs is to experiment more and more and learn from it.
For the Zoe depth, I agree with you, I've been testing with the depth in control net also, it turns out to be an extra helper in shaping the composition. So yeah it's good.
@@jerrydavos Thanks for the reply! I'm going to share a couple of my findings.
I also noticed that the prompt can have negative effects on the final image. For example, sometimes the image would be over-saturated in red, or have too much black, and so on. Even though these colors were supposed to apply to very specific things (red hair, black pants, for example), the colors would often "bleed" outside of their intended target.
I have found a potential solution to this problem. An extension called "ComfyUI Cutoff". According to the creator, you can specify parts of your prompt and "cut them off" from the rest of the prompt. (I'm a terrible explainer). Basically: it would ensure that "black" only applies to "pants". "Red" would only apply to "hair", and so on. In my testing, it seemed to help to some degree, though more testing is needed.
I have managed to make a near-perfect smooth video, converting a real person into cgi/3d render. From what I've seen, if you want the exact same person/character/outfit (but still influence some things), Zoe Depth / Lineart work best, with OpenPose as a third option. For some reason, Zoe Depth is much more consistent than OpenPose for minute details like arm/hand positioning. OpenPose has a tendency to get a little... confused as to where arms/hands are (front, or behind subject). These will still allow you to change small details like clothing/hair/skin/eye color.
Tile works great, but only if you want an exact replica of the original subject. Otherwise it can be a real fight to keep it from "intruding" in the final image.
I have started to experiment with replacing a character completely, and I did manage to succeed, with one specific character (more testing to be done), and I'd even go as far as saying it came out really damn good. I managed to keep the background under control,
Again, thank you very much for the reply, and the tips! I had no idea you could raise AnimateDiff's context... For some reason I thought that 16 was the maximum.
Ya, I agree with you Zoe depth can a lot compared to open pose. In my next project I will try to implement depth pass a major support.
And as for the Comfy_cutoff node, in theory it should have worked but didn't (I compared it with my old render and still no change) with the animatediff loader or say with the temporaldiff model as the author said
"Current limitations are that the labelling for my dataset was a bit off, so it has slightly reduced ability to interpret the prompt, i'll be releasing a new version that fixes that soon."
Maybe animatediff loader model is the issue, so it has to do something which is causing this color "Bleed" I will test on other motion modules as well.
If you don't mind, your render videos sounds interesting, can you give a link to it or message it on discord (jerrydavos), I am very curious to see what you have achieved so far.
Hey! I totally missed the notification for this message.
I'd be happy to share some of the videos I have generated so far, if you still want them.
I have videos from when I first started up to now.
And I agree with the Cutoff node. It kind of worked but kind of didn't... I've experimented with a lot more things since then, and have managed to get some good consistency on some clips but of course, still trying to overcome certain challenges.
I've added you on Discord. From aelemar
@@danielo9827 I have read your tips bro
Can I see your videos too?!
And see your workflow?!
Unfortunately I don't have discord account
that kid in the end credits is so cute
I can't get same result as you. For the poses of control net its okay but I get a weird and horrible background. Is there any solution for this kinda problem?
Lower the Batch range to around 20, lower batches has the highest details, in longer batches the details gets blended and thus becomes a smudged background.
and also try to change model, some support longer batches some does not.
Epicrealism_naturalSinRC1VAE, Aniverse, animerge is best for realistic rendering and they support longer batches. and Meinamix for stylized.
You can see the Bug List note inside the json file if it matches your problem
In the dancing videos, the outfits change throughout. But, at the end, the lady in the gold dress does not change very much at all. Is there a way or best practice to keep the character looking the same through the whole animation?
The greater the screen coverage, less the change.... you can try the v3_mm_sd motion model for lesser change... It's still experimental to get good consistency
how do i use a lora along with a checkpoint model? I ve tried using lora loader after load checkpoint node but any lora gives me horrible results in vid2vid in comfyui
some loras are not compatible with AnimateDiff, you have to experiment and test which one works
I wonder why you did not put the Preprocessor nodes in the main json and just did it in a seperate json? Is there any performance difference?
Yes, for low graphics cards Ram is a major issue. It kept getting out of memory error. Plus It make the testing a whole lot faster as control net does not take the extra 2-5 seconds every time to get the pass. So yeah Performance gets better.
Can you do this with any video file? If I was to say download a video of someone dancing from TH-cam and then use that as my starting point would that work?
You have to do the procedure and it will happen eventually
❤❤
I want to know if Stable Diffusion can also make such a rendering work？What is unique about ComfyUI?
Yes, A1111 also has AnimateDiff, but It's take a lot of time setting it up again and again and sometimes animatediff does'nt work with controlnets. It's in development for A1111 we have to wait for a proper stable version.
In comfyui you just have to drag and drop the workflow no need to setup again and again, this is unique about comfy
Great tutorial, btw there is something I don't understand on batch results.
If I make a preview of a 20 frames batch the result is a set of specific images. Without changing anything in settings, with same seed, if I extend the batch to the full duration of the frames, the result is completely different (and definitely less interesting). If I render in batches of 20 frames skipping 20 frames each time, the results is visually better but each 20 frames the images changes so it's useless.
Do you know how to fix this?
Suppose:
Batch 1: 1-50 frames
Batch 2: 45-100 frames
From frame 45-50 fade in the opacity of Batch 2,
5 frames rendered extra for cross fading so it becomes a smooth unnoticeable transition,
Total Output images 105 ( 100 source frames and 5 overlapping extra frames)
I've used this overlapping frame technique here : th-cam.com/users/shortsErXBU5WD0ZY?feature=share
In your case, 20 frames gives the best result so you can render 2-3 frames extra and cross fade them in post. At least it will be less noticeable.
And this is thing which will improvised in the future of Animatediff.
@@jerrydavos thank you for your answer but I already tried the overlapping technique but the result is really messy, it changes each 17 frames and sometimes the change is so drastic that the fading (or morphing with other AE techniques) is completely useless.
Use lineart pass with high weights in controlnet along with openpose if its a human video, else use HED plus Lineart with 0.8-0.9 weight
The problem is, this ai animation technique is new and still in "Experimental Bases" So you have to try various combinations and steps to get your desired results.
It will be developed in the future.
@@jerrydavos yeah I know, it's always a matter of hundreds of tries
Hi! it doesnt show the "loadimages" node .. its just there the "loadimage" for only one file .. any idea? thx for the tutorial!!!
just solved updating comfyui "Update All" :D
7:00 " I have 3070 Ti laptop GPU" and this is the reason why you start to work on comfyui, guessed by someone who have exactly the same GPU😅
3070 Ti 8GB laptop was the best I could afford in 2022 😅
would their be a way to impliment this stick figure refrence pass as a realtime video recording for say full body tracking in VR. cuz i have used programs that do this kind of thing that mimics imu trackers and it does the whole base skeleton thing but this seems much more accurate idk if it has to do with the fact it has at black with gray outline effect i would think that would help the tracking of the skeleton much easier cuz it would make the process easier if the algorithm knows whats moving and what isnt and i dont believe the VR version does that. it would be fascinating to do because currently full body can ring u up to over 800 bucks at the least 176 with 5 imu trackers using slime vr trackers, so this would be a wonderful free alternative that might produce some pretty good results.
I agree with you, it would be very usable for the mass without any mocap rig. There is still some issues to the "Skeleton" OpenPose tracker that it does not work well with body rotation yet, which is back view which is front view, it renders always the front view, which make sometimes weird results. This openpose technology should develop more in the future coherently using passes like depth and hed pass for figure out accurate pose, Then we won't be needing any mocap rig. And this skeleton can be rendered to anything you can imagine.
i am a begginer and not sure where to start do u have any videos you know that i could watch or any tips
First watch this video, it has all the basics:
th-cam.com/video/z-AoELaJfn0/w-d-xo.htmlsi=kqpevdofHlNmyizm
then explore on these, I learned a lot from these channels :
1) www.youtube.com/@promptgeek
2) www.youtube.com/@OlivioSarikas
3) www.youtube.com/@sedetweiler
4) www.youtube.com/@Pixovert
With this work flow - you can not use XL versions? Is there way to change the face to an animal? or does it have to conform to the pose of the face then?
1) Unfortunately, I can't test on SDXLs as I don't have the Hardware and SDXLs motion models are still needed to be trained properly.
2) The faces Can't be changed with this workflow, Here might be something that can do like that : th-cam.com/video/4826j---2LU/w-d-xo.html&ab_channel=KoalaNation
Hey question beginner here i got some decent results with comfyui and stable diffusion. but how do you change the background in ADetailer, i cant find a way to do so. do you have a solution?
Adetailer is for Face Detailing, you can't change background with it, You can find workflows to change background in this site : openart.ai
@@jerrydavos i found a way too do it, extra's remove background, then merge and invert mask in ADetailer. then mask erosion on 16. still have to tweak but it seems possible. have too work on it some more.
The transformation rate matches gen z’a attention span
Hey man, i have a problem with the animations part at minute 6.00. When i start the test of 10 frames of animations, in controlnet 2 nodes becomes red and gives me errors. This 2 nodes : control_net_name: 'control_v11p_sd15_softedge.pth' not in [] || and : control_net_name: 'control_v11p_sd15_softedge.pth' not in []. How can i fix this??
You need to have some basic beginner experience in comfy in order to use this workflow easily.
The error you are telling is of controlnet models not present, you have to download them from here huggingface.co/lllyasviel/ControlNet-v1-1/tree/main
and place them in their proper directories. (ComfyUI\custom_nodes\comfyui_controlnet_aux\ckpts)
and please use my latest workflow from here : www.patreon.com/posts/update-v2-1-lcm-95056616
it's much better and I will be uploading a tutorial on how to use it soon.
Hope your query is resolved !
@@jerrydavos hi i have fixed yesterday night, thanks for your answer! Unfortunately i cant use this software because my notebook have a rtx 3060 with 6gb vram..there is a way to use it with 6gb, this is a disaster 🙈, when all seems working, it goes "out of memory"
Can bloggers provide an installation package for these plug-ins? There are many problems with the configuration environment and installation problems.
you have to explore tutorials on youtube
Thank you sincerely for making this video and your hard work. I have two question.
1. Is there a way to decrease the randomness? I gave as prompt to wear something black. The girl i used starts with a white top and ends with a black lingery and in between changed color a few times. I am using the same settings in this demo video.
2. What dimension input do you recommend to create a 1920x1080 video?
3. Stable Diffusion: I can't select negative_1
1) You can try the V2 version of this workflow which creates more stable animations : www.patreon.com/posts/update-animate-94523632
and for the randomness, it's part of how the AI animation works in animatediff, if it was not morphing, the animation would become choppy. it has its cons.
2) Try with lower resolution in the raw animation workflow file first - around 1k resolution then upscale it in the refiner workflow to around 1.5k resolution for best results
@@jerrydavos I get the memory error (1st json). My video is 1920x1080 - 1min long. What do you advice? Thanks
downscale it to around 1280x720 using the settings inside, longer than 20 seconds crashes the save node. so Chop it into 3 x 20secs videos ....... or even 15 secs if it still crashes.
@@Paranormal-yg4ph
Thanks, that solved the problem. I'm now at step 2 but I am stuck at step 5 "Paste the Passes Directories in ControlNets (1 and 2) Loaders and set their models and weight (See Neon Purple note)"
I don't see a purple note near it and I can only input a directy. What directories should I point out? With the 1st json file it generated 5 directories: Canny, Depth, Frames, HED, Line, Normal, OpenPose.
Thanks again!
@@jerrydavos
I feel you are new to comfy, You need to have some more experience in using it. This workflow might be an advance stuff.
For Face Closeups - Use Line Art and soft edge.
For Half Body Shots - Use Line Art with Soft edge or Open Pose.
For Full body shots - Use Open Pose with line Art or Soft edge
For Architecture or landscape - Use Depth with Soft edge, line art or Canny.
For Flat Logos, or Typos - Use Normal with Line Art, Soft edge.
Check if open pose passes, are all over the place jumping around, jittering markers, or cannot find important body markers, then avoid the open pose pass.
Two Control net passes are enough to make a good Render. You can experiment with any combination from the above.
@@Paranormal-yg4ph
Hi, Jerry, could u tell me which url about Load Image From Directory node of ComfyUI?
github.com/Kosinkadink/ComfyUI-VideoHelperSuite This is the one
Thanks for creating this tutorial. I'm not sure why I can't get it working. I use the same workflow file with the sample given HED and openpose images. I even use the same models that used in the workflow. I got blank (black) images when I set the batch range to 10. and then I tried to set the batch size to 1. I got colored noise and just barely see the shape of a human.
Your Model or Lora is not compatible, Try epic realism model or mistoon
no luck with these models either. I suspect it is because i'm using a macos with apple silicon.
ya maybe@@sisilet
thank you However, as shown in the video, an error occurs while processing kssampler while running. Can you tell me the reason? comfyui NoneType' object has no attribute 'shape'
Lora, or Some Model might be missing... Please See this tutorial for installations : th-cam.com/video/qczh3caLZ8o/w-d-xo.html
. Also Do Not use SDXLs models or SDXL loras in this workflow, It not compatible with it
@@jerrydavos I solved it thanks to Gower. Your skills are really great.
Why is the image coming out messy even though I followed the workflow you provided?
Sometimes, Models and Loras makes it messy, Please watch this Video for more info : th-cam.com/video/aysg2vFFO9g/w-d-xo.html
Help Please! I use your workflow (or in A1111) when i use softedge contlolnet my resaults always red-ish and washed out blury. WHY? yours are great....((
Please check your controlnet models are properly set.
If you are using comfyui inside A1111 (Comfy extension) this workflow will not work. Use standalone Comfy
Hello, problem here ! When I load "3) AnimateDiff Refiner - LCM v2.1.json" I get an error: "When loading the graph, the following node types were not found:
Evaluate Integers
Nodes that have failed to load will show as red on the graph." , I don't know where to get this node.... help please !
It's Efficiency-nodes-comfyui, it's no longer updated by the author, so it gives error.
Evaluate Node error can be solved by Deleting the already present efficiency nodes folder inside custom nodes directory and manually copying and pasting the v1.92 in the custom_nodes folder from the following link and restarting Comfy as administrator to install the remaining dependencies : civitai.com/models/32342?modelVersionId=156415
@@jerrydavos Hi, Thanks 👍 It worked and I moved at the 4th step, but then again it ask for some 6 Nodes the Manager and I (manually) am not able to find : UltralyticsDetectorProvider, SAMLoader, ImpactSimpleDetectorSEGS_for_AD, SEGSPaste, SEGSDetailerForAnimateDiff, ToBasicPipe
github.com/ltdrdata/ComfyUI-Impact-Pack
This is the Github for the Node, "Run as admin" before running comfy to install all the remaining dependencies @@gibs2b
I have tried a lot of workflows, but always the video changes drastically every 2 seconds (every 16 frames). Why it might be the case?
the motion models are trained on 2-3 seconds of reference videos (16 frames to 32 frames) so for now it can handle only upto that. This should develop more in the future.
Wait is this only possible with Comfyui? Can automatic1111 also do it? I just ask cuz i see very few auto1111 tutorials for this type of workflow and its not even the same workflow
Yes you can do it in automatic 1111 also, at the time of release of this video it was not possible. Now you'll just need a good vram pc to run on A1111 for longer videos though
is the ADetailer in A1111 better than the FaceDetailer in Comfyui?
1) A1111 is simple which is good for beginners but it can be used only one time per image,
2) On the other hand face detailer needs experienced user but can be manipulated to our desires and can be used multiple times.
Friend, I'm lost, there are many nodes that I can't install from missing nodes, what version of comfyui should I use???
No need to worry... just use the latest version, with all the latest custom nodes.
Also use the version 4 workflows from the Gdrive folder for best experience
@@jerrydavos Thank you very much friend, I will try it again, your tutorials are one of the most complex that I have seen on TH-cam, my question is, is your method useful to change the style of the source video? Or can I also reinterpret the same pose from the source video but with a different character like with loras?
After moving to automatic1111, You only showed your adetailer when adjusting the face. Your denoising strength was at 0.75. I’m assuming you had controlnet enabled. Can I know what controlnet you used exactly
The Img2img denoise value is skipped (disabled )... It's not affecting the image, only Adetailer is taking place on only the face
I've check the Skip Img2img box
@@jerrydavos okay thanks
07:08 how do you know, whats the calculation, how many frames can you batch generate according to widht, height and your gpu per batch
it's calculated by observation,
just render in the first try if it shows out of memory error or not, increase resolution gradually till you see the out of memory error, then Note the "Requested" Memory if its in GBs then you need to lower the reso , if its in MBs then you are close to the Maximum limit your PC can handle.
When 300 - 500 MB Requested memory then you need just little decrease in the Resolution by 10% then you should have you Max limit
Same goes to the Batch range,
Maximum Threshold = Reso + Batch Range
When it goes above this then Out of memory error.
Hope this make sense .
Is it possible to generate a matte to separate the FG and BG later?
Yes It's possible with this extension
github.com/biegert/ComfyUI-CLIPSeg
What are the minimum requirements for a GPU, Ram and processor to run this version?
RTX card with 8 GB Vram
I got KSampler error whenever i use controlnet, what problem could it be?
Error occurred when executing KSampler:
'ControlNetAdvanced' object has no attribute 'model_sampling_current'
Maybe you are connecting wrong inputs and outputs, also check if you controlnet models are properly placed in the right directory or they might be corrupted.
Else you can send your workflow on discord : jerrydavos or drop the link to the workflow in the comments.
I'll try to help you.
@Ai_Davos Thanks for quick response man! But i just reused your workflow to modify a bit. All i did was only changed the to dreamshaper model, change the image size and the 2 path for the controlnet folder. You may take a look: drive.google.com/file/d/17T9Ntjqv8KbJ0eMUyXPcRdZTxyWnXRvj/view?usp=drive_link
nevermind i fix it by reinstall everything. Thanks!
I downloaded these three workflows and used the second workflow to complete the openpose and hed two sets of pictures of my 13-second video. Everything was normal. However, when I used the third workflow, as the video said, I Tested 10 pictures, and the results of the output pictures were all noisy pictures. The models and parameters were all the same as the videos. Could you please tell me what the problem is? How should I solve it? Thank you in advance, looking forward to your reply
Then I tried to increase the number of sampler steps, from 22 to 30-50, but it had no effect. The final images were all noise images. I thought it might be a compatibility issue, and then adjusted the large model many times to no avail. Anyway, I checked. There were a lot of parameters, but I didn’t find a solution to the problem in the end. I was a little bit broken, haha
1) Models and Loras produced these noisy artifacts, try different checkpoint model like epic realism and remove any loras
2) Try simple prompts with no weights...
3) Try Removing all Negative embeddings in the negative prompt box, to see if it changes anything
4) SDXL models won't work with this workflow, this workflow needs to be converted to SDXL workflow first
You can watch for more info here : th-cam.com/video/aysg2vFFO9g/w-d-xo.html
Thank you very much for your reply. You are very helpful in listing so many possible situations.I'll try,respect~@@jerrydavos
I've tried all of the above, but this workflow doesn't seem to work for me. Finally, I turned off the controlnet node, and the background was still full of noise.@@jerrydavos
Please save your workflow in json file and the passes you have exported for using in the control net in a zip on a google drive and forward me the download link on discord. Username : jerrydavos , I'll test it and see what's the problem @@MrJi666
Thank you for the great tutorial. Unfortunately I get this error when executing, and I'm not sure how to get around it. : Error occurred when executing KSamplerAdvanced:
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
If you have any ideas, that would be great. TIA
Try setting the cuda device to gpu with this argument "--cuda-device 0"and try again
you can also use this updated workflow for better results.
www.patreon.com/posts/update-v2-1-lcm-95056616
Thank you @@jerrydavos
Well, those video Stable Diffusion AI are still very umm... unstable
after reinstall controlnet aux node for many times,still get a "Dwpreprocessro" not found error, do u know why?
Maybe some permission error from inside comfyUI so it can't download properly.
Manually delete the comfyui_controlnet_aux folder from the Custom nodes in comfyui Directory if present
Run cmd from inside the Custom node, go it address bar in the explore and type cmd and press enter then
git clone github.com/Fannovel16/comfyui_controlnet_aux.git
Try Running the cmd as administrator if it still can't download properly
Hope it should get fixed, and download rest of the control net models from here : huggingface.co/lllyasviel/ControlNet-v1-1/tree/main
Thx for reply,Still not work after clear the folder and reinstalled it.I replace DWpreprocessor not to "openpose recognition" node,press queue botton,it gives me a "Error occurred when executing OpenposePreprocessor:
No module named 'matplotlib'."The control net models is ok, i can see them in load controlnet node.@@jerrydavos
@gvg0105
Option 1)
I found this link: github.com/Fannovel16/comfy_controlnet_preprocessors
which uses the matplotlib, try to git clone this one too and restart comfyui to download requirements.
Option 2)
Go in comfyui folder where the venv folder is located. Run the cmd there and enter these commands :
- venv\Scripts\activate.bat
- pip install matplotlib
You can "pip install [missing module]" like this from cmd
Option 3)
If still not working, go in comfyui, download and paste these folders inside ComfyUI\venv\Lib\site-packages
drive.google.com/file/d/1U1gUwHb_V0UkgBupPoO9CuixDikUZ_26/view?usp=sharing
Backup the venv folder just in case.
Option 4)
If it still doesn't, rename or move the venv folder from comfyui directory, and start comfyui, it should make a fresh venv folder and download all the dependencies again (It's gonna take internet data in GBs )
It should be resolved now else please send full error log
I found these is no folder named"venv" in ComfyUI_windows_portable\ComfyUI, then i create one use python,and tried option1,2,3,4.really comfused@@jerrydavos
[comfyui_controlnet_aux] | INFO -> Some nodes failed to load:
Failed to import module dwpose because ImportError: cannot import name '_c_internal_utils' from partially initialized module 'matplotlib' (most likely due to a circular import) (F:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\matplotlib\__init__.py)@@jerrydavos
You would have to make a zip with a comfyui already configured with the nodes, it is really a problem when you cannot add the nodes from the common list, more than anything because it may be that they are incompatible with some other node that you do have in the list ,
I'm already past installing nodes and having comfyui die,
I even currently use 2 different versions, one that I update all the time and another that is a little older, from when the net control was working well, (things that do not work in the new one), to make matters worse, both versions are increasingly larger in space, (at first glance I think that with the old version and the net control that if it worked for me I may be able to do something similar, I know it is a tremendous amount of space, but I ask: is there any chance of taking your comfyu configured like this, making a copy without the checkpoints/loras, so that it does not spread too much and upload it to the internet with the nodes? Experience tells me that each comfyui is a world, and they can have more than one incompatible thing, personally I am doing very fun things in comfyui, but no I have a way to upload the complete comfyuis so that if I make a video, it works for everyone, I have even reached the maximum limit of nodes! Anyway, it would be greatly appreciated if there is that possibility of uploading it somewhere, if I just look at my folder controlnet weighs 22gb :S, I'm also running out of free disk space, but making room and downloading something slow even if it is very large is something that could be done, they are ambitious projects, but it should even be possible to use a single outfit, and character, without that varies at random, although for this it would require more nodes
If at some point someone from comfyui answers me about how to add more, I will make myself dance to that but with a clingon face xD
Greetings! If you think you have the chance to make a summarized backup, it would be greatly appreciated!
When I encounter something like this, I rename the original custom_nodes folder and make an empty one and run comfyui and install the Missing nodes with Manager, and restart 1-2 times the comfyui to download the dependencies and check if all are working fine, then I copy the New custom nodes in the old folder and restore back the name and not overwrite anything.
Yes, I understand you problem, I also had the space issue for different versions, You can try the Link Shell Extension, I have all the models in one folder and I copied the symbolic link in every versions of a1111 and comfy, it's a trick to have multiple copies of same files without it's actual file size (it's type of a shortcut link with original properties) since then I was helped a lot by it in also everyday works on my PC.
I have 2 x A1111 and 1x comfyui and it occupies space only for 1 of them, just the Ui files occupies the space.
Hope this helps!
ValueError: only one element tensors can be converted to Python scalars
Can someone help me understand this error ?
You just have to delete the Dynamic Thresholding node from the workflow and it will run fine, without any errors. It got bugged after the recent update.
Link for photo reference: imgur.com/a/9bBNPDo
anyone having an issue where they run the images through automatic1111 with the adetailer but none of the images are being saved anywhere, even though an output directory is specified??
If the output directory is not working for you then just don't write anything in it, Copy and paste the face fixed images from the A1111 img2img Output folder
This same problem also happened to me when I Didn't made a output folder inside the explorer first. I thought A1111 will make a new directory itself but it didn't.
what is your gpu or computer specs to run all of this
8GB RTX Laptop GPU
32 GB Ram
Error occurred when executing KSampler: only one element tensors can be converted to Python scalars. how to fix this error ?
I've never encountered this error though you can try these, but it seems like a python error:
1) Try deleting "FreeU" node from the Model pipeline, it won't affect the workflow.
2) Make sure you are not using SDXL models inside this workflow.
3) Re-install matplotlib through pip commands (Search google)
4) Python version should be 3.10 or above.
If the above didn't solve the problem, ask for help on Stable Diffusion Reddit page about this python error : www.reddit.com/r/StableDiffusion/
You just have to delete the Dynamic Thresholding node from the workflow and it will run fine, without any errors. It got bugged after the recent update
Link for photo reference: imgur.com/a/9bBNPDo
I cant find HED lines please help , theres only HED Preprocessor provider
Search for "Hed soft-edge lines" or Hed preprocessor ..... the node is renamed in comfy
Also install controlnet nodes
ksampler is not showing any preview like yours. what is the solution ?
github.com/pythongosssss/ComfyUI-Custom-Scripts
I think this was the node... It's been a long time , forgot which, It was enabled by one of the custom nodes from the description
I still don't understand how the frame number is set, how to operate the set to skip frames, can I make a video here
It's like a pizza system, you can't eat it all in one go, you slice into pieces and eat one by one.
Similarly, Video is sliced into batches so PC can render them one at a time.
And after all batches are done, it is glued together in the end
Thank you for your patience, I tried to set 0,50 yesterday; 50100; 100,150 and so on to the last number, but the resulting picture is more than the original frame number a lot of pictures, this is right@@jerrydavos
you may have accidently re rendered same frames, in the skip frames just add "+50" before every batch and it will do the math automatically @@user-yb5es8qm3k
Author What were you dealing with on AE in 14 minutes 20 seconds? I don't really understand
Is there any way to use it without gpu but faster? Like using virtual machine or something?
Comfy also runs with CPU but you can't virtually install RTX card, No.
@@jerrydavos ok. Thank you so much for your reply
Why is it that when I output 10 frames, it turns out a different picture, and when I output 50 frames, it turns out different?
How can I check the picture before full launch?
10-20 frames has the maximum details, above that (like 50) the background details gets smudged, AnimateDiff has a limit how much it can render frames in one go, so I use a trick to render 20 frames and stitch them in post while having a overlapping cross fade.
I have given a breakdown of this in my next video : th-cam.com/video/aysg2vFFO9g/w-d-xo.html
@@jerrydavos thx
@@jerrydavos I have one more question: when I generate an image into text 2 image, the picture turns out rich and juicy. And when there are several frames, it turns out somewhat dull. What affects color and how can it be changed?
How to do in stable diffusion? Cuz I am using in my phone on Google colab so it's so small screen to use comfy ui
Sorry, It won't work on google colab.
@@jerrydavos I mean can you make a video for Animatediff img2img on automation1111 stable diffusion
please drop the link you used fix the face ......stable diffusion link
It's automatic 1111... to run stable diffusion and Adetailer is a plugin of A1111 to fix the face
Do you have a google colab link for it?
Sorry I have not used google collab, in theory the json file should be working inside the comfyui google colab version if it possible.
it works!!
can anyone help me with the error : "Error occurred when executing ControlNetLoaderAdvanced:
PytorchStreamReader failed reading zip archive: failed finding central directory "
Copy the path of the controlnet images folder and paste it in the CN directory node... it should go away.