Make your own ANIME with this new mind-blowing AI TOOL! (ComfyUI Tutorial + FREE Workflows!)
ฝัง
- เผยแพร่เมื่อ 26 มิ.ย. 2024
- AI movies are finally controllable! I show you how to combine my consistent & poseable character workflow with a new tool called tooncrafter to create completely consistent AI animation.
If you like my work, please consider supporting me on Patreon: / mickmumpitz
Follow me on Twitter: / mickmumpitz
You can find the FREE WORKFLOWS & INSTALLATION GUIDE here: www.patreon.com/posts/new-vid...
You can find my PATREON EXCLUSIVE advanced guide, example files and poses here: [coming very soon!]
Check out my last video on this subject: • Create Consistent, Edi...
Tools used:
ComfyUI github.com/comfyanonymous/Com...
midjourney www.midjourney.com
tooncrafter (huggingface demo) huggingface.co/spaces/Doubiiu...
tooncrafter in ComfyUI
Chapters:
00:00 Intro & Overview
01:00 The Tool
02:00 midjourney
02:37 Tooncrafter on Huggingface
03:04 Using Photoshop
03:37 Create a model sheet in ComfyUI
04:19 Posing the Character
06:34 Animating the character
08:02 CleanUp & Upscale
09:05 Reference to Anime
This will work best when you can draw the poses and fill in mismatch frames your self also you can use exaggeration anticipation, animators are getting upset but actually they are the best equipped to make content, their works will be so much better than the non artist prompters, I hope more of them experiment and push this,
what animator has been upset about this
cringe, this is for larpers
@@bake-io1cf 50/50.yes , but also can save time if you are an animator and want faster results (for fan animations, or for fun, just draw keyframes, color them in, and don't have to color each breakdown and inbetweens, (only for simple animations like headturns, smaller movements,etc.)
It'll be great for like low budget shows but yeah this is not gonna work in a studio as of now .
@@tr1pod623 if you were a professional this would only make the end result worse without actually improving efficiency at all
Damn! You are really pushing the whole nascent AI animation industry forward! Been following for a while and every video you come up with a more sofisticated approach! Congrats!
Animators have been using digital tools that speed up the workflow for years, so I don't see this as a bad thing for animators. For example a very popular tool in the animation industry would automatically fill in vectors for in between frames roughly and then the animators would fix the mistakes the software made and then move on. So I feel like anyone complaining about potential AI tools for animation probably aren't actually in the animation industry at all.
This is amazing. I am an artist and therefore wouldn‘t have an issue with those messed up frames or keyframes, the ai didn‘t do so well and just draw those myself. This just speeds up time so immensely. I always wanted to create an animated short film but the amount of work, I would have needed to perform, simply was way too much for my disabilities. With such tools and workflows I have hopes again. AI tools are such a godsend and help removing barriers. Either the barriers of not being able to draw or paint or whatever, or like in my case the barrier of partially paralysed hands, chronic pain and a severe fatigue. These are just amazing times.
What exactly are you trying to make?
Did you try this workflow ?
Smartest kid on the block! I mean it! Your workflows are always so innovative and clever. Thanks again for another great demo of what can be achieved to create animated films.
Thank you so much!
amazing work. thanks for sharing!!
That's it, I'm moving over to Comfy
I think I'm about to do the same. And I hate Comfy...
Yeah, this video made me do it as well
Most amazing work i ever seen.. great job on this one.. 💪
amazing thank you, ill be sure to look into this soon
Nice job!, thanks for sharing
love your content. Always something interesting ❤
really coolll!!!! I love it!!!thanks for sharing!!!
One of the best tutorials I've seen in a while. Thanks a lot for creating all of this and sharing literally everything with us!
wow tht´s sooo cool!!! Thansk fot sharing!!
Super impressive! Really appreciate the information. Your workflows are some of the best I’ve used, if not the best.
amazing work
Your doing some Genius level SHIZZZ bruh thank you
Thanks for making this video🙃
Wow amazing!
thank you for making these workflows free, you could have easily pay-walled all this information so i do appreciate it much!
Once again, amazing workflow, working smoothly for low gpu. you're the best
I was hoping you would release this one ! Just started watching but it looks promising 🤩🥰
You know what, I am going to try this. Mess around with it a bit. Might not be perfect but the possibilities for creating proof of concepts with this intrigues me. It would give me something to do with all this writing that has built up over the years.
Beautiful ❤
so nice. tks for sharing
This series of workflows is great! I have an anime-style music video that I want to create, and I thought it was probably going to be another year before the quality and temporal coherence of these tools reached an adequate level. This isn’t perfect, but it’s close enough that doing some touch-up in Photoshop becomes viable. Thanks again for figuring all of this out!
Thanks for the info
Dude your intro is epic straight up
Ooooh and tooncrafter is apache license. Nice!
Not convinced the amount of work to set this up is worth it. Whats really getting me is the difficulty on posing the character. When you make a pose and render the image you are not guaranteed to get what you wanted, complex pose gets messed up with that 3d poser as to depth maps for the hands... thats alot of work for each and every pose you may want. When you can take that 3d poser and make 3d animations as you would in maya and then be able to apply any character to that reliably, and be able to tweak timing of animations add in or remove frames, then I think you have something. Until then keep up the good work.
Sibal... It's really amazing
ai lip sync + ae face tracking would be perfect to the intro with the characters talking
Yeah man you are really smart, thanks for share what you neuron cook with us to cook better!!
Finally we can fix the battle between Escanor and Meliodas
First off, thank you so much! You made it so simple to install and to play around with it.
My first results show up as images without much animation between, character moving lips then jump to the next image. Any idea what I may be missing here?
Reporting back after more testing - So I managed to get the animation to come out! Just chose images that look more similar to eachother. My first tests were too different key frames so it didn't even try to animate between them. I would like to experiment to try and 'force' it to interpolate even if the keyframes are very different (like getting one character to turn into something else)
Didn't have much luck with the upscaling workflow probably because I am not using an sdxl model. Any idea how to adjust it for sd1.5?
Amazing
thank you for all of the cool workflow ideas! if possible, do you maybe know how we can feed the workflow with an existing image of a character, for the creation of a character sheet? The PulID and InstantID models are not cooperating well with anime style characters as an image source
awesome
I seriously need to relearn how to use comfyUI soon :3
Great Video, I love the mixed media approaches you are working with here, its fantastic. One thing that still stands out to me is all the steps in the sausage factory you still need to get results. I beg to question why wouldn't someone still work in a more traditional animation manner to create something without maybe some of the quirks? Regardless, the final effect you achieved is awesome, I just feel like if you're young watching this and diving into all this tech, maybe also learn toonboom or maya as well. It will only serve you well when mixing with comfy,etc..
the camera movement, if the camera movement can remain the same, or controlled, for example stop it from zooming in/out or panning (unless speceifiec) the animation would be better. Another thing, animation smear, I think the keyframes that one puts as an input if they are too far apart causes more noticible animaiton smearing. Smearing is nice if used correctly, specially in higly dynamic action scenes, but in calm slow scenes it looks off. I think maybe adding either more keyframe for slow scenes, maybe not more keyframes but psace them aout accordingly. its gonna be an iterative test until we figure out how much 'time' or 'frames' is best to put as reference
How do you handle multiple characters at once with these methods?
While it's useless as a final product, for those unfamiliar with animation principles, particularly timing, it can be helpful to trace over the animation to fix consistency issues. For animators, it can speed up workflows, like figuring out the in-betweens needed to adjust the animation speed (for your sanity, keep it on twos, don't try to do 24 fps in ones with this process XD).
I'll definitely not use Tooncrafter in its current state, but I'll keep an eye on it for sure.
Thanks for this tutorial!
Make an anime show using this workflow
@mickmumpitz could I insert a IPAdaptor somewhere in this workflow to input a pre-existing look? If so...where
Add MuseTalk to your workflow and you have lipsync to finalize it. Always enjoying your tutorials!
Does anyone know which custom node the set_vae, set_clip etc. are? The manger does not find those when trying to instal missing custom nodes.
Genio!
i really can color an BW animation. how many correction in the color is needed?
can we run this on google collab? please can you check that?
Yes
Hey all, I am a game developer and I stumbled upon this video. To make a 2D fighting game like Mortal Kombat or Guilty Gear, one could potentially create the model sheet in ComfyUI and then then the various poses in Open Pose AI. Then, in Unity you could just create the animations, VFX, juice etc. to get it all working and feeling like a completely original fighting game. Is anyone interested in connecting on Discord to go through this and collaborate on a project like this? I have about 4 years of experience working in Unity for a few games that are live and in production. I am really interested in diving into this, but would love to do it with a team!
Got this error when executing the tooncrafter workflow, it happens when it passes through tooncrafter interpolation node: "Error occurred when executing ToonCrafterInterpolation:
No operator found for `memory_efficient_attention_forward` with inputs (...)"
Have you tried ipiv morphs workflow for something like this, I'm pretty sure you can get something similar and more smoother
Wow, you are smart an you know how to use alle the AI tools toghether
Make full anime show using this method.
Excellent video, thank you for sharing.
My art style is kind of like Invader Zim. I wonder would it be possible for me to create a model sheet in that kind of style, and then shoot my own video reference footage.
I would like to think regardless of model style it should still work well.
Hope to experiment this weekend, and I'll find out I guess
How much vram i need for toon crafter workflow?
Well that was complicated. I wish this could all be done in one application.
Also, I’m still waiting to be able to use AI to create scenes with ‘multiple consistent characters’ interacting with each other.
Would be cool to have this as colab workflow
Hey man, that's amazing, would be great to see a short animated story via this tool. Any plans for this?
Can you explain more on the upscale thing? Make it simple
explain how i can use it to color sketch animation
Great work! I think you forgot to add installation guide!
Working on it, give me a few more minutes haha
@@mickmumpitzOh, I am so hyped for this workflow🤩
我想问一下在哪里可以下载或获取姿势参考。谢谢！
Which hardware did you use?
RTX 2070 SUPER, 8GB of VRAM. If you have something better you will be able to create even better results! (higher resolution upscales for example)
I was with this until about half way though then it became a lecture in advanced computer science....
What's the minimum RAM requirement?
Show the process for coloring BW line animation
dont u need 32gb vram?
i tried tooncrafter demo and for some reason it doesnt work as good as advertised. any tips?
I also tried it with the example images and it worked very well. But for some reason it didn't work with my own drawn images.
If I use ipadapter and animatediff for interpolation just like this instead of tooncrafter, will the videos be more smoother, and will it be better?
This also works very well and was also my original idea for this video. However, the animations were less precise in my tests and the characters looked slightly different. I also like the option of adding an additional prompt in tooncrafter. In the example of the waving woman, it not only interpolated between the two poses, but also created new waving poses for much more natural movement.
I don’t have pc it’s possible to use ipad
Unsolicited advice: I think you should probably get a real computer if you reasonably can? I don’t mean “a *nice* computer”, just a *real* computer. A computer that doesn’t artificially restrict what you can do with it.
An iPad is super locked down.
You don’t have to use the command line, but if your computer doesn’t give you command line access, it isn’t a real computer. (This isn’t a definition, just a heuristic.)
To answer your actual question:
I don’t think an iPad has enough computational power to do these kinds of things. I don’t think a powerful enough (already existing) GPU would fit in such a small thing (along with the other parts it needs)
Earlier in this comment I talked about getting a “real” computer, which I distinguished from a “nice” computer.
To run these things locally, I think you would need a computer which is both real *and* somewhat nice.
Potentially you could remote into another computer, or like, send the other computer computational jobs to do, and have it send back results, and this is something you can probably do on an iPad.
It’s possible someone offers a paid service where you could do such things on their computer, *through* your iPad?
What the anti-AI crowd don’t understand is that AI is simply a tool, it is not replacing artists or creators. But really at the core of most AI hate is that it is another tool that democratizes an art form that was had an expensive entry point in addition to a steep learning curve.
I'm getting less of that from the artists than the fact that generative models were trained on their copyrighted works under an assumption of "fair use" that is not yet legally tested. There's some gatekeeping, but most of it is that they're not getting paid for the use of their work to train the AIs in the first place.
There's too much black-box stuff going on. Bloggers are complaining that some models are live-scraping their sites, disguising their User Agents and ignoring robots.txt, and have proved it. OpenAI is not open about what was used in their training sets. It'll take another 3-5 years for any cases to progress to SCOTUS for a definitive ruling of what constitutes "fair use" for training for-profit AI models.
Essentially, too many creators of gen AI models are "getting rich" off of potentially illegally training their models to copy the work of artists whose works are being used for training without compensation. Even Adobe, which claims to work with artists for Firefly, is being accused of appropriating art off Adobe Stock for training without honoring an opt-out.
The biggest concern is whether you'll be able to trademark or copyright a character created with Gen AI. Last I read, the Library of Congress stripped an AI comic book of its copyright, then issued a new copyright just for the story and layout, but the art was not allowed to be copyrighted.
How much does quality prompts work, like I see "masterpeace" and "best quality", but how can the AI know what that is when those have to be very personal, and then "4K, 8K" that made me giggle a bit for they ask for two different things and how would AI be able to create an 8K image within 1025x1024? And what award was it winning?
My question is, can you remove those "quality" prompts or replace them with random things like "324k", "Nobel prize winning image", "lame quality" to see if that make the image uglier?
its a plagiarism app for larpers
Using ComfyUI requires SD right?
However, *SD just changed its terms and conditions.* (For the worse)
Prompting employees to leave / quit?
Not sure if ComfyUI has a lifeline being coupled to SD.
Thoughts?
Are the terms only for newer models? Like, did original license allow redistributing under the same license?
but overall it's just so extra might as well just learn to animate, and then just save time without having to inbetween
Cool to see the workflow. I don’t hate AI but the end result still looks super janky to me. Only the shot with the painterly girl is sort of nice but it’s extremely limited. Can see this useful as a rough Animatic tool. But as an Animation tool, I feel like it generates too many yucky frames that would take more time to fix than just drawing them correctly in the first place. As of now, I could only see this making believable results in the hands of real experienced animators. Makes me wonder if AI is always going to be stuck at cheap fastfood levels that never gets elevated or improved over the years, like McDonald’s. I doubt this tool could ever be used to make a full blown Ghibli inspired animation.
CAN WE DO A SHORT FIGHT ANIME WITH IT
You should test that stuff always with a bit mainstream Pokémon/Digimon/Pal - since those are not very prominent in the training data and therefore shows you the limits of this faster. Avoid mainstream and waifu Pokémon
💪🏽💪🏽🔥🔥🔥
what he doesn't tell you is that you need allot of vram and that will most likely only work for people who can afford a a40 gpu, the only way to run tooncrafter is with a runpod on the cheap
Nope, not true. My workflow is optimized for 8GB VRAM. Everything you see was created on a 2070 Super RTX 8GB Vram.
@@mickmumpitz my bad, you should let everyone know that its possible to on 8 gigs vram, also you should try using ebsynth to make it even smoother and flowframes even more smoother like butter
its too much complicated ... but looks cool
must the art be AI-created or can all the source art be created by an actual artist?
I think u can use any art
awesome brother
ive been on it for about a week i have a dope workflow that creates some great art
"art" lmfao
@@j.2512
@RiiahTV
0 seconds ago
@j.2512 what does art mean to you? if you dont mind me asking you the definition. to me art is how humans use tools to express their thoughts. i believe people like yourself fall in love with the tool itself. and forget that at one point the tool that you use for art was once looked at like it wasn't. remember this "there once was a finger painter in a cave, a man came with a paint brush and he hated how it didn't do what his hands could do. there was a painter painting a photographer comes and takes a picture of the woman. the painter didn't like how he couldn't control the camera like he could his paintbrush. the photographer was taking pictures a man with a reel begins taking multiple images to make film. the photographer didn't like how he had to put together images to make movement. the videographer and the photographer are working together now and a guy with a computer shows them photoshop. they call it blasphemous. it's not real photography. a guy that uses videography and photoshop is doing his work when a person that uses an animation program shows him what he's capable of without any physical tools like a camera. the guy with the cam like real footage and really getting out there to capture real settings. the animator meets ai. the animator embraces change because animators know when new tools are introduced it only means progress in art. young blood remember artists use tools to create art. just because your not smart enough to use a hammer doesnt mean some one cant grab a chisel and make an ice sculpture of mona lisa. wake up are is everything. art is perspective. art is life. you are a blackhead on the face of idiocracy.
Hey didn't comfyui have a spyware thing recently? Or am I thinking of something else?
Thanks for pointing out, I didn't know this. Looks like applebotzz keylogger is now spotted and on the blacklist. Update comfyui.
You're kind of correct. ComfyUI allows you to install additional custom nodes created by the community, and one of those nodes had spyware in it. I always focus my workflows on the most tested and popular custom nodes created by trusted community members. However, there is always a certain amount of risk (with most AI tools that you install locally).
Brother better you use Krita whit confyUI plugin than midjourney.
use morph transition not visibility😄
Always love your vids @mickmumpitz even when you were making sketchy blobby 3d chars. There are similarities to the way sketch animation on paper works, but the ai is filling in the blanks
Imagine how much this will help artists who can just fix a bit the main animation and get a masterpiece!
I totally agree! There's no reason Professional animators have to exist now. 😁
you suck
I thought we were a year away from something like this. This is incredible
Anyone else lost when it comes to the vary regen tool that MJ should have? Because I don't see it. Anywhere.
Dude! For me, this was like the 'Holy Grail!' Bless up, good sir! Thank you!
for the untalented hack is the holy grail
WOW..! Cool..!
Id waste less time actually learning to animate 😂
If the guide is free why pay to become a member, confused af
Wait wait wait. You mean to tell me...I could draw key frames....and this can create the in-betweens based on my illustrations?
....Well, let's get CRACKIN'! O_O
It doesn't have to interpolate full-colored illustrations, just lineart.
Your work was sensational but somewhat inefficient. Certainly, using just the style transfer technique for a live-action performance with some post-production editing would be better
I applaud the effort, but the workflow is a nightmare, and there are key limitations that makes this unsustainable for a serious attempt on an animated narrative.
If the same workflow is to be used, for the best output, I’d recommend having a 3d character and 3d environments.
And again, if the character were to interact with an object or kiss someone else for example, the ai may find doing such tasks extremely difficult at the moment, borderline impossible.
I hope this gets better soon. This could be very useful.
it looks bad though..