TrackAnything - ComfyUI Segmentation (with ControlNet, TemporalNet and masks)

💪 Support this channel with a Super Thanks or a ko-fi! ko-fi.com/koal...
☕ Amazing ComfyUI workflows: tinyurl.com/y9...
🚨 Use Runpod and access powerful GPUs for best ComfyUI experience at a fraction of the price. tinyurl.com/58... 🤗
☁️ Starting in ComfyUI? Run it on the cloud without installation, very easy! ☁️
👉 RunDiffusion: tinyurl.com/yp... 👉15% off first month with code 'koala15'
👉 ThinkDiffusion: tinyurl.com/4n...
🤑🤑🤑 FREE! Check my runnable workflows in OpenArt.ai: tinyurl.com/2t...
============================================================
Video animation made with Comfyui 3.0 - Stable Diffusion.
😉👌🔥 Run ComfyUI without installation with
ThinkDiffusion: tinyurl.com/4n...
RunDiffusion: tinyurl.com/yp... 👉15% off first month with code 'koala15'
The goal of this animation was to be able to keep as much consistency of the colors of the clothes and the face features.
Consistent masking in ComfyUI, despite the different tools available, is difficult for batch to batch animations. Therefore, I have used a modified code of TrackAnything (the original did not work properly), to identify the different segments and create masks from there. TrackAnything allows to review the created masks and correct them on the WebUI quite nicely. So at this moment I would recommend to use this tool.
The masks created are used in ComfyUI to change the colors of the top, the jeans and the hair color of the dancer. An overlay over the original frames are made, which are later to make the frame sequences.
A standard animation with ControlNet (Lineart, DWPose/OpenPose and TemporalNet) has been used (check my previous tutorial to see how to use it).
A character Lora (Rebbeca Chambers) has been used to keep the consistency of the face. Results are quite consistent for most of the frames. FaceDetailer has also been used to further refine the face features. However, when the face of the original frames are not totally in the picture, many artifacts and deformations appear. Similarly happens when the dancer tries to turn around: the position of the arms and the body are not correct.
I have tried to correct some of these frames by re-masking some of the wrong faces with the face from a different frame, with improved but not very good details (despite using FaceDetailer). For some isolated frames, frame interpolation has done the trick quite well (sequence of 1 or 2 wrong frames).
For the rotoscoping and video effects, I have used Runway.ml, and for the final video editing ClipChamp.
CheckPoint: RevAnimated v1.2.2
civitai.com/mo...
Lora: Rebecca Chambers (Resident Evil) LoRA
civitai.com/mo...
ControlNet: LineArt, OpenPose
huggingface.co...
TemporalNet (diffuser)
huggingface.co...
TrackAnything (original):
github.com/gao...
TrackAnything (modified):
github.com/dsi...
AI Animations
Released: 13-Aug-2023
© 2023 Koala Nation
#comfyui #stablediffussion #animation