Turn your AI images into CG movies [Workflow Video]
ฝัง
- เผยแพร่เมื่อ 19 มิ.ย. 2024
- This workflow explores how to use various AI tools to create compelling character animations and "render" them using AI.
If you like my work, please consider supporting me on Patreon: / mickmumpitz
Follow me on Twitter: / mickmumpitz
Tools used in this video:
- midjourney: www.midjourney.com/
- ChatGPT: chat.openai.com/
- blender: www.blender.org/
- ICON: huggingface.co/spaces/Yuliang...
- Stable Diffusion WebUI: github.com/AUTOMATIC1111/stab... Control Net
- Sad Talker: github.com/OpenTalker/SadTalk...
- Deforum: github.com/deforum-art/sd-web...
- Adobe Creative Cloud: www.adobe.com/
- DaVinci Resolve: www.blackmagicdesign.com/de/p...
-runway GEN-1: research.runwayml.com/gen1
- Wav2Lip: colab.research.google.com/git...
Other resources:
- Sad Talker (Huggingface): huggingface.co/spaces/vinthon...
- High resolution Depth Maps for Stable Diffusion WebUI: github.com/thygate/stable-dif...
- 3D model Male Figure: clara.io/view/d49ee603-8e6c-4...
- Blender Beginner Tutorials:Blender Beginner Tutorial - Part 1: • Blender 3.0 Beginner T...
The ULTIMATE GUIDE to Cameras in Blender (Beginners Start Here!) • The ULTIMATE GUIDE to ...
- Stable Diffusion Models: civitai.com/
Chapters:
00:00 Intro
00:32 Creating the characters
02:16 Generating the 3D model
04:03 Voice
04:26 Facial Animation
05:58 Body Animation
07:51 Patreon
08:25 "Re-Rendering" with GEN-1
11:44 "Re-Rendering" with DEFORUM
14:12 Results
![พาแม่บินครั้งแรก..เกือบวูบ! ให้เกาหลีเยียวยาใจ I Korea Ep.1 [Seoul] x อุงเอิง](http://i.ytimg.com/vi/waL3__dEQBU/mqdefault.jpg)
I have an idea to make it better. When you create character's 3d model Focus on detail of characters head and face . Put it into unreal engine 5.2 and Resolve Metahumen identity. After that Use your iphone to best facial capture. And use mixamo to body animation. I think you will love that workflow .
You can use a still shot of the character's face with EbSynth and make it so the style doesn't change at all while they're talking.
I tried that, but it didn't work very well because of the low resolution. EbSynth often has trouble making Lipsynch look convincing anyway, and the results I got looked better and were less work. But I will take another look at ebsynth (especially Tokyojab's workflow) for another project!
@@mickmumpitz Good to know. Did you try just using the first frame of the character, which was the original image?
That’s amazing!
It’s interesting that we learn the tools, but there’s always something new on the way you use them that’s completely different and does more than we expected. I am definitely becoming a patron today!
Thanks a lot for sharing!
I am amazed by how consistent you got Deforum. If it wasn't for some of the left over flickering, I would prefer it over Gen 1. If you combined it with Tokyojab's workflow for Ebsynth with Stable Diffusion, I think you would have knocked it right out of the park (for free, but a lot of work though). Thank you for all the work you're putting in and how well you document everything. Always look forward to your videos. Amazing quality! Keep it up bro!
nice to see you keep watching new workflows all along mix em and then share it with us
Holy moly, that intro trailer is just incredible! Fantastic work.
Amazing work! Congratulations man
This is insane! AND it will only continue to improve
Oh, it’s so amazing😮 wonderful job
Man, your videos are truly amazing. I'm just dipping my toe into AI video, animation, and editing, as I'm working on a proof of concept for an 80s-style creature feature. I will definitely track your successes as an inspiration/method for this project. Thanks 🙏
absolutely stunning video as always
thanks so much for sharing. your description is easy to follow and your edits are entertaining to watch. great job.
thanks friend. i always wait for your videos
Great stuff! Your workflow videos are clear and also relaxing. Thanks for exploring these processes and sharing withe the void.
Definitely agree that animation and even filmmaking will eventually include an "AI pass" at the end. Similar to color correction and sound mastering. So cool.
This is all happening so fast. Excellent videos!
amazing tutorial! thanks for sharing your expertise!
Brilliant work!! I totally love watching your videos, thank you for sharing and can't wait to see what else you have coming!!!💯🔥🔥🔥
Great job man! Thanks for all the hard work of creating these videos.😊
😅
You are crazy! This is awesome and what I've been looking for! Very impressed here and thankful
Yeah wow. I think this is the future too, and it's awesome to see your prototyping at this super early stage of the game. I imagine "in the future" there will be programs dedicated entirely to this process that are as ubiquitous as photoshop. It is your tinkering now, that paves the way to that future. I am going to sub to your patreon to get you a new graphics card! Poor thing must be melting! Thanks for doing what you do.. I am on the same journey and have basically been devising the same workflows as you, but thanks to your work, I don't have to test them all out. Its awesome and I can't wait to see the film. Much Love from the land down under :)
Outstanding! thank you for your hard work!
pretty cool.. keep up the great work
You are cray, man! great job
Thank you for this tutorial. ❤ Very helpful.
Phenomenal video. Thank you.
Sehr cooler Workflow, vielen Dank für's teilen :)
great work mate , keep up
Thank you for sharing!
The lip sync idea for the texture was sooooo fucking smart!!!!
Thanks for this informative video👌🏾
It's really refreshing seeing someone diving deep with all the current tools and go through workflows on a good speed. No Mumpitz here, your channel is a little gem, very inspiring! Runway ML is really milking it hard, thanks for not only sharing all the commercial solutions like most others.
tremendous work!
Damn so dope dude!
Thx a lot ! I’m working on an ai series on TH-cam and your video will save me a lot of time! Thx again ! Great job!
Incredible display; what an exciting time for independent creators!
Bester YT Kanal seit langem! 💪🏽🫶🏼🤝🏽 #1000thx for sharing und die Einblicke in den Worklflow… #ehrenmann
AMAZING！！！LOVE！！！
Thanks for sharing. Love your bandwidth. And I also agree, that its always best to consider supporting the open and free software, like a1111 or deforum as we can all grow from it! rock on!!!
The best!😮
You make everything sound so easy LOL
just saw the intro and thought: mega!
Thanks ❤ your other videos a few weeks before where very much Inspiration for me. Ive tried and learned extrem fast. Now i am almost on the point that i can compose my whole pieces from all these amazing tools i used (yes also payed did and wonderdynamics, runway....) i am a hobby creator and a single dad of a 7 year girl. 6 Month before i only got a smartphone and cuttet some Rapmusicvideos of my own Hobby(parent)-gang. And now i am on a point that i turned my created avatar into inreal engine. Looks great and moves 😊. I see how fast this development improves workflow and getring Ideas come to reality. Ive not uploaded so many stuff but i would be honored id you check these few vids. As i watched your videos i haven't even heard the word depthmap, but was so impressed by your ideas, that i am annoying anyone in my life with things they don't and pretty sure won't hear about it. My big Project is called "The point of convergence" A fantasy storie which is about a fictional me and the world of my friends. And the (try and error but learning)-project is an Androidstory(oh so creative😂 i know😅). Besides i step into learning comicanimation with cute characters for my daughter. I produced a learning video used free version of adobe character animater, canva pro and filmora 12 pro. Think will upload it these days on my second channel because it stands out pretty much from my Homechannel. For my android Story i used did to sync, but spoke it myself(and friends) with VOICEMOD pro and recorded and finetuned it with Reaper and IzotopeNectar (which is the only programm and vstplugin i was familiar before i stepped in this crazy mindblowing ai world😅) feel free to visit my channel. You can also see my beloved little daughter in her first role playing in front of an Greenscreen almost 2 Years ago. (No panic at all Webhoodwatchers: playing a role, coverd in her acting as this person and facecoverd because i am not a ****. Looking forward to see your next stuff. Because if i see these voluminous golden hair and hear that smooth English voice, that i was teached in german school, i know i will be illuminated with knowladge for weeks. Not sure why i dont take a translator or an ai😂 so excuse my mistakes in writing (on smartphone) or grammer. The "idocrazy" way of life is possible near, so i think use our brains so long as we are allready know that we want to use it by ourselfes. Good Night good Life. Have to rest its 03:27 am😢
thanks a lot !
Wow! Grossartig!
I love this video - what would improve it is a list of softwares you used, in order, with a price for monthly subscription for each. As a hobbyist trying to forecast if this process is a worthy investment, the numbers would really help. Thanks again!
You are so talented man, you can use so many software and I'm still struggling with fusion in davinci resolve
The neck movement crack me up 😂
genius!
learning some new shit today👍
i use the Quad Remesher addon for blender (like $89) to get better topology on the character mesh, then I use SoftWrap with a custom face/head mesh, to redo the head. then we can use another addon I haven't gotten yet that generates blendshapes for the character in blender.
Thanks for sharing your workflow its is so creative and really useful. The final result May not look pixar, but it is on the way, lets say to a 65% which is unbelievable Because Just one person can do it in record time. This is the future man..
cool techniques ¡¡
Someday there will be a single program that will do all that
Thanks for ai❤
This is amazing do you think I can use the same software and techniques to create a music video?
U make everything look so easy to do... thanks, been looking for exactly this, how do u make ur own animation using AI. might need AI to take over my brain to do this anyhow, but I'll give it a try.
wow!
Sehr sehr sehr cool!!! Danke für die tolle Tipps, ich schulde dir, big time! Wenn du willst, dann möchte ich dich ein paar Runden auf Bier kaufen haha
Lovely video, thanks for sharing! I've been meaning to test reprojecting the high resolution texture back onto the lip sync after wav2lip with Ebsynth but haven't got there yet. Would love to know if anyone's tried this and/or encourage others to.
I also have a concept for a node setup in DaVinci Resolve that will track a face and export a cropped image sequence for external editing (for example, working with Stable Diffusion or Ebsynth with smaller files, but maintaining higher local resolution)...
15:05 well... I think it's pretty close! And i also think that Pixar and companies alike are already exploring these possibilities. I mean, why not? 😃
Hey, that was absolutely amazing. Thank you so much for providing such value. Do you also teach over Zoom and offer one-hour classes? I would definitely be willing to pay for this. Could you provide your contact information so that we can discuss it further?
Respect for exploring what the tools can do but there seems like so many steps I wonder whether a professional animator / CG artist could achieve what you're doing at the same standard (if not higher) in the same amount of time?
That is what i am planing to do..... sad but happy someone done it before me
I also suggest to use 2k2k rather than ICON cause 2K2K is better on skirt.
And, it seems cloak and hat is such a chanllege in 3d modeling...
extra-ordinary ! mickmumpitz what a research ! as story development pro I can see the possibilities but this video results shown - at this point -suggest that it would be a lovely 'bad transmission' video to actors within a narrative. perhaps cutaway scenes in videogame - for instance: a 'lost video diary'... 'its impossible' has a too far extreme head movement that an animator would not do. the old man movements got better... but I guess tom cruise is not at risk today. aiming pixar at first shot could possibly be a bridge too far?
but I believe! animating a full feature animated w your take on AI is just a bit further away - very possible [ but screenplay upgrade is a must ].
in my opinion trying to do ALL animation with AI now is going to make a result but not a longing delicious animation.
if you tried to make a longer teaser scene using more camera work it might be less strenous to watch because the visual cinematic narrative would compensate for the visual weirdness noise. unfortunately a book from XIX requires extra screenplay adaptation and animation skills.
on the other hand, check animation from Nat Film Board Canada has a very high level of arts experimentation, and some as Coe Hoedeman took opportunity from low developed characters to tell a very sophisticated story. why not try a funny monster that only has a noisy line to say ?
another aspect is that animation is an art of transformations, so a person that shifts is likely to succeed as storyline.
meanwhile, cheers from Rio.
you said would like to see what we do: Bandolero ! short 5 min live action cartoon th-cam.com/video/OgckykwLjMg/w-d-xo.html I've used a cartoon 3d background +chromavideo into unreal.
Could you please tell if you used the original output from blender directly to deforum and it gave you a good lip sync, without need to use all previous instruments to fix lip sync?
This is amazing! You just made me realize that I can save thousands of dollars for my project, I never realized that it was possible to do this on my own, thank youuuuuu 🥰😀
They guy have a Patreon running, time to share if you saved a hundred dollars, right?
@@ManuelJimenez1 I didn't mean it like I already had the money for my project, I meant it as like "I don't have to wait to have a way of funding these details in my project to start it". I'm a student lol, I only have enough to survive, but I was already planning on subscribing to his Patreon page, because he gave me what I needed to be able to continue my project, because I was seriously about to drop it. Anyway, it's cool to see that you already subscribed to his Patreon, that helps support him 😊
@@ManuelJimenez1 Also, can I respectfully ask why is your comment so passive aggressive? I don't believe I said anything bad.
great! have you ever tried nvidia omniverse audio2face for lipsync?
Thank you! Yes, I actually did in my first video "How I generated a full CG Short Film with AI".
What would you recommend for creating animal models? I need rats, spiders, Akbash dog, etc.
when the AI processors are powerful enough the graphics cards will have two stages the 1st will generate a low poly image and then the 2nd stage (generative AI) will generate from this image a high resolution image with the style of an artist and in real time
Did you check "runway GEN-1 alternative"?
Hahaha you are the man.
Does ICON work on a mac
No idea what's happening. I would've used Vegas for the lip-sync battle... so prolly outta my little pond of expertise. That said... FABULOUS! Something there for everyone! See? Why can't banks be like you... just giving away cash-knowledge...
How many did it cost for the total procedure ?
Hi How about do you use CSM to make back of character?
Csm is great, but the quality of the characters is unfortunately not comparable to Icon yet. But I already use it for many other 3D models!
ooo i wonder how much gen-2 would improve this.
i made 3d models in daz studio, we can use them in blender using their ears and some hairstyles skipping all those steps
hello friend, I have a few unpublished and unfinished graphic novels, and I would like to learn how to create animations with AI to bring my graphic novels to life. I would like to know if you had consider to do private classes, how much you would charge. Regards, Clau.
I am very interesting animation programming but I don't know idea animation program software,
Before I sub?
Can you share how many hours in TOTAL the entire workflow took to finish this 15 minutes Vid?
Oh, what the heck!!!
Sub'd and Liked 👍
For all the work YOU'VE 💪DONE!
4:49 free tool for ai face animation
This is what I was searching. Thnks
Need advice how I can make a character still if I'm in a wheel-chair. I would definitely struggle to get into that T-Positition... Would love some advice please? … Kyle x
It’s still quiet far from everything. There are already better non ai tools that can do all this extremely well. It would take people to work specifically into developing software just for the purpose to mix all this together and make it accessible. In other words, people with a large pocket that can afford to use money in high risk high reward investments.
So for the next three years, it’s going to be an all out war until all this ai software can communicate well with stablished or new edge applications. The sooner the better.
Icon doesn't work anymore :( so can't even follow this tutorial haha... :(
heheh Looks like Tom Riddle from harry potter. (his 16 year old version)
Imagination is dead.
Oh, hope your toothache are better now....
Fix this fix that.. urgh..
looks may deceive
why not just sculp by hand, in the end
HOW do you know all of this?? AI has only been in business for 15 minutes!?
my opinion, is horrible, and not efficient yet.
Yet... :)
Your content is great, but the clickbait thumbnail is off putting. The content is strong enough to stand on its own.
not so good final result
TOO MUCH WORK. need an ai which can do that it one click. lool
Or you could just use stable diffusion...
Man….in the end, after all those hours of effort and money, I have to be honest and say: it looks like shit :/