Convert Ai generated 2D images to 3D models for use in Blender and Gravity Sketch.
ฝัง
- เผยแพร่เมื่อ 16 พ.ค. 2024
- This is an overview of how I use Ai generated images from Midjourney and Stable Diffusion to create 3D models that can be used for design reviews and refinement. I cover the 3D conversion process and how I manipulate and use that 3D data in Blender and Gravity sketch to help create 3D forms for further design maturation.
Use this to convert 2D images with depthmap estimation:
huggingface.co/spaces/shariqf...
You're good at figuring this stuff out. You're on the forefront! Please keep sharing.
Very cool workflow. I think the clean up in regards to the warped texture bits and the missing back and bottom are really tough bits to work out. Using depth maps like this is a fun technology and tool. Thanks for the vid!
pro tip: in blender if you set the fbx "Path Mode" to Copy and then click the little button next to it to embed textures, they should arrive properly in GS
AH thats what I was looking for. forgot the embed texture. Thanks!!
FINALLY somebody using gravity sketch!! thank you this is a game changer!!
5:00 Adding an up to scale character model or a face to the scene would make adding depth and fixing the size much easier. There are a bunch of free character models, just check if the character size is around 185 cm (or the equal of the character collision for default characters in the engine of your choice). Simply put the cut side of the helmet over the character's head so the eyes and eye holes meet and then mirror and scale what is needed. This way, you will use a single character model (size) for all of your projects and it would make it easier in the long run as well...
Agreed. I did this in gravity sketch so when I build the final geo I have that reference. its a scan of my head so I can pretend to wear it.. until I finally make it.. 😅
@@JamesWRobbins It is a good idea and an amazing method to save you some time! I sadly do not have a VR set, so I could not play around in the same way. That being said, I am also a lvl/ environment artist so my approach is naturally different. Scale how ever is one of the most important things regardless, something I had to learn the hard way once I started working in engine many moons ago XD
My new favourite chanell - Looking forward to more Gravity Sketch♥ & AI smart optimisation solutions!
Fantastic work. thank you.
jeez 4:42 a select lasso for vertices omg never knew that thanks - and thanks for the rest too wow! amazing man!
Interesting workflow! Doesn't look very practical, but interesting nonetheless. I wonder if we'll still have physical monitors a few years from now, or if it's all in VR or AR.
Awesome workflow! Thanks for sharing 👏
What a beautiful tutorial! haha Thanks man
Very nice!!
Thanks!
this is actually the FIRST ai workflow that I've seen how ai is intended to be used. we need more content like this. I just wish the model's geo was a lot better
The dev just needs better math implemented like smoothing edges like adobe does and a feathering option which doesnt seem too hard because its just vertex math basically. Like if the dev trained the ai to use those maths, it would be better
Just wait until it can do proper topology and fill unseen spaces. This one is a more "hacked" way of doing it, but already looking good. Not practical without decimation thou
@@pebegou wont happen.
@@jeffyboi6969 It not only will but it already does. There are AI retopology and shit that do tries to complete unseen spaces. The way this dude made it has zero input from AI to generate mesh. Also the ones we do have now is pretty bad for now, but with enough training one can even fuck yo mom.
That's amazing
great video thanks!
You can take this and import into zbrush , add some guides and then z remesh a duplicate. This will clean up in quads but you'll lose your uvs. Now take the other duplicate and flip the uv map vertically , assign the texture from the model and divide in order to convert to polypaint , you will be able to store the diffuse map in vertex color info instead , bake that out of substance painter and work from there overlapping more maps and normals as well as perfect the geometry all this in blender. I like this workflow very cool
Thank you, this was incredible.
Thanks for the video mate! I am having trouble enabling the texture, when i click the top right drop down menu, i cant see the same options as you, i see less options (blender 3.5 btw). Is there a way I can make all these options show up or other way to turn the textures on? Thanks a lot!
Really interesting stuff, thx I'll surely try thos workflow
great stuff James!! Zoedepths is awesome!
This is great stuff! Thank you.
Nice workflow, thanks!
Its a fun workflow but i feel like the traditional modeling workflow might be quicker this over complicates the process of building simple geometry 🤔
But this also includes the generation of the idea. This illustrates how you can go from nothing, not even a concept image to a 3D model in about 15 minutes with the help of Ai and a little bit of manual work. In the future that will be the workflow.. This actually greatly simplifies the process. You don't need multiple views for reference, and then you have a 3D volume to build over with texture reference include on that volume. You would just have to do some retopology and cleanup. Depending on the final output needed, obviously the workflow for this will vary for Game asset to 3D print to full manufacturable part. But I'd love to see someone do a concept render and build a textured model in 15 minutes..
The ai idea generation is great my main qualm is with the 3d generation part from 2d to 3d, you still have to work hard to create the proper looking peices of geometry that the 2d to 3d software didn't get right enough for you to do retopology on. However This workflow could work for me if I use the 2d to 3d generated model as a base mesh that I'll sculpt and polish up in a sculpting software like zbrush and maybe remodel and replace pieces I don't like then combine it with the new sculpted mesh then finally retopology 🤔
Exactly. Its a good starting base because of the volume and texture detail
I am conflicted. The purity of hand modeling vs the reality of ai/ai involved worklows. But i got to agree. Minutes versus hours is invaluable. I don't wanna fight the technology advance. I want it to fight for me..👍
There's no way you can traditionally model this helmet faster than he did this process....
10x mate!!
Though I wonder how do you select ALL the vertices in the Lasso, it seems like I delete and I still have vertices remaining in that area.
I’m a stylized 3D Artist and watching this makes me happy because now I can truly see my work flow is much faster and cleaner.
awesome workflow!
(5:21) You should do two things first:
a) use a plane or cube to "subtract" the fuzz on the back half of the imported model because there are a tonne of vertices you don't need that will bog down your performance, and a subtract will give you a nice tight edge.
b) scale it on the X (or Z) axis first because most of the proportions will be preserved, compared to "pulling" on the mask will cause small distortions you'll have to iron out later.
Yeah, I wasn't sure how much of the "Fuzz" would be used so that's why I use the cube to just hide it. Once its deleted I can't get it back so that's one way to "delete" it and still keep it in reserve. Not worried about performance with this mesh. Some might, so its a good idea.. Each model is off in some axis but it varies so I just pull where I feel its needed at first. its just a rough in so I'm really just going for a volume and texture with this for reference as a rebuild later. Its just a proof of concept at this point..
Adding a few primitives like a half sphere to the top of the helmet
This is amazing , never knew you could use this shortcut.
Still a proof of concept and a little work but not bad for 15 min of work
ГЕНИАЛЬНО!!! ЭТО ШЕДЕЕЕЕЕВР!!!
very coincidental i came across this. i just generated a mask on midjourney the other day and have been trying to figure out how to turn that into a 3d model
very useful for experimentation, for now
Fantastic tutorial.
Wow! Very cool and interesting work!
This will sound mad but is there a software I can add a pre made 2d animation and ai will convert to 3d or am I 10 years to early
great tutorial
thanks, quite helpful for abstract effect haha
Hi James, I was just wondering how the results came out. Did you 3D print the helmet? Did the model come out good as expected? I'm curious.
As Blender beginner, i like seeing the keys u press.
Then, you have to do retopology. Using ZRemesher or alternative. Displacement modelling
Great video, I was wondering is it possible to flip the 2d image in order to get the other side of the image so it less manual work that has to be done??
Cool and interesting
its crazy how you can now edit in VR
very good one .thanks
I'm wondering what's ur prompt to the helmet, it looks so perfect!!
hi thx for the video its awesome could you save this blender 3d file dxf format thx again
wow thank you so much dude
nice work
Nice!
Good workflow concept, though I'd probably try to generate a front view and a side view of the object and then use a more traditional blender modeling approach to get a game ready object.
or do retopology of this and not have to guess at the volume. But depending on the final need there are many ways to make 3D models 😀. this is a POC/ experiment on possibilities.
Good tool! l am preparing for my new resin printer. Ai concept to 3d model to physical product.
I'm not a 3D artist but I watched some of Nvidia's recent developer conference and they showed off some tech that automates this sort of process to some degree - taking 2d images into 3D space. I think it's called "Instant NeRF" or something like that. Cool to watch you do this. First time I've seen someone work in VR space. I've been generating some really novel forms with stable diffusion using 'sacred geometry' and Platonic solids, and while they're only like 80% coherent I feel like they could be a jumping off point to creating some fascinating 3D art or sculptures.
yeah its really nice to see my dream job getting automated...
@@METTI1986LA Is your dream to make cool 3D models and artwork? This is just a more sophisticated tool in your pipeline. There will need to be a human in the loop for a long time to come.
they just wanna complain instead of innovate@@parkerault2607
Hi, good job! not possible fbx with texture ?
With this I could make some pretty cool 3D printed cosplay stuff.
What would be the best way to do this for a full body mid journey 2d character image to 3D model? I have a character that I love for a new animation, but only have one front-facing image of the character since I didn’t get the character design sheet while prompting in Midjourney. I can’t find any other answers to this online, so any help would be greatly appreciated 🙏🏾 thanks for this tutorial btw it was mad helpful
th-cam.com/video/Kf9oSHRWdNg/w-d-xo.html
Wow do you do this for a living? What glasses are you using? Do you have any recommendation if just starting out wanting to make AR Games?
For example, I like TiltFive (aka CastAR) but I'm interested in more FPS and with more standard AR glasses (Google Glass/Hololens 2/Lenovo ThinkReality A3).
BTW, any thoughts on Luma Labs' Video to 3D API) , Blockade , Spline AI?
There are no good AR products released at this time, the best you'll get right now is the meta quest pro which is really built for VR not AR (this is what I assume he is using in this video), or the meta quest 3 releasing later this year.
4:32 I dont manege to delete the parts?? It delete only small parts of what is selected?? What did you do?
Hry this is pretty good
cool video✌
can we run zoedepth locally? and where do suggest i start for learning blender?
what a time to live...
What you're missing is a better depth modes with multiple view angles and automation via the huggingface api.
Honestly, the forefront image at 0:00 should have been what the First Order helmet in Star Wars should have looked like. It's intimidating, looks like a natural progression of helmet design, and consistent with the Star Wars lore. Of course, minus the orange.
in which year you are living!!!! you are way ahead of everyone
New era.
3D generate AI to 3D Model to 3D Printing?
That's the plan. Working on that now..
when i go to the link it displays a "Run Tim Error" and a long list, why is that?. but thanks for shedding light on this tool, absolutely magnificent.
Not sure, maybe a huggingfaces issue. its not my tool so I can't control anything with it..
Awesome work - next step would be retopology back in Blender ie. trace the model to get something more final.
This might be ok for a quick concept but I don't think it's anywhere near as efficient as a normal workflow for creating something useable at the moment.
Making a depth map from an image like that is basically just going to give you a blobby mesh lacking any kind of detail. So to be useful for anything it will still need a lot of sculpting and retypo. Someone competent at scultping could create that rough blobby mesh faster with just some simple scultping.
On top of that the images from AI are going to have all light and shadow baked in because they are not albedo maps. That means they are noy ideal for textures anyway.
I think a better workflow is to just use AI to help generate ideas and then use those ideas as reference for your modeleling and texturing and just do it the normal way.
I'm sure at some point in the next few years AI will be able to generate 3D models as well as we can generate images now but until then I think traditional workflows are still the way to go imo.
Nice video anyway!
So basically you have to sculpt the helmet, ai only helps in projecting a texture, you dont need ai for that
A tip for you dont delete the noise at all in adobe audition or any audio software you use that's make the voice looks weird
Do you do these projects for other if requested.
This is sick. Had no idea this was possible
I wonder if it can work nicely with my drawings to convert them in 3d
Does it works with face images?
what hardware do you have?
yes
finally.
Fuck yeah!
This is great. Specially for us beginners .
That helmet gives off helldivers 2 vibes
Impressive result! I'd love to see some of this AI muscle brought to bear in the service of an AutoRetopo Addon.
That would be a great use case for sure
Hi, I looked at your app and unfortunately it is not functional at the moment? How can I help so I can try it out?
Not sure, maybe a huggingfaces issue. its not my tool so I can't control anything with it.
As a 3D artist, I'm still not quite satisfied with the process itself.
But we definitely feel that the technology of 3D creation through AI has now made an insignificant start.
Until a few years ago, such an attempt was unimaginable.
I look forward to seeing how much more the creation process can be improved in the future.
good job
Agreed, This is just a start and a proof of concept but I'm sure it will become more streamlined and functional really quickly
this is great, though some retopology and edgeflow work would be needed after, this would shave off hours of work in the beginning
"In the name of progress, and most importantly (!) -profit!; a pronounce The Death of Creativity ! "
It's a hard-surface model. Could we use the method to generate 3D character model?
i would not sugest useing this for characters. cuse how it works details would be messy, like the hands the face over all. ect . and thats if you have a straght on A or T pose. if its a pose like crossed arms then youed pretty much have no arms or hands at all depending on the pose.
@@the_stray_cat I hope the next version of Midjourney can do that
good for drafting and conceptualizing but i wont submit this quality to a client as final.
But wait, technically it'll work with non-ai concept arts too no? The possibilities are scary with this approach
I have a question, How can I export as a render if the material doesn't show me? That's the negative thing...
Can you do this with a photo of a person?
PiFuHD was the best image to 3D I've ever tested, generating a whole model from a single photo. Unfortunately, it only covers human shapes.
I agree, I had gotten some great results from it also.. There's a lot of new tech papers out so I suspect we'll see something useable very soon..
@@JamesWRobbins Having a great image to 3D tool would be great for hi-res polys because retopo already makes our lives miserable XD
That would be one less problem to deal with.
much efficient is to design it from scratch all together in blender,
Imagine it would be possible to generate the other views matching in AI so you have more reference images to generate the 3D model
that is kinda possible to do now
@@kamel3d how ? I did not figure that out yet
@@cekuhnen You can *sort of* do character turnarounds in Stable Diffusion, look into it.
@@j_shelby_damnwird yes I saw that with characters - but does that work with objects too?
@@cekuhnen I've been out of the loop of AI for a month or so (ages in AI-tech development pace) and as far as I know this was strictly for characters. Who knows? maybe you can get consistency in object turnarounds now!
Instead of diffuse node, try emit node when you bake, thats one way to assure only the color data & no light data bakes, without needing to mess with many other settings.
Good tip, Thanks!
for me the grabber isnt doing anything
monster mash works better than this imo, at least for simple structures. i just wish that the lines you draw in monster mash could be drawn one by one so you can allow for mistakes and if it could automatically make a border around non transparent part of transparent pictures.
After seeing the wireframe of this "3D Model" I really relieved as a 3D artist. With this mess of a wireframe, AI will not take over my job soon...
Sorry but beside some basic still renders a mesh with that topology can't be used for anything...
The sad part is that people with no knowledge of 3D art will use these horrible meshes in games and then wonder why things aren't working properly
Thank god somebody wise in comments, and the most sad thing -beginers will watch this dogshit process and will do more same million triangles helmets ))
AI is gonna do all this in under a second soon
Basically absolute dogshit 3d models, thats great! ))
Awesome tool but it doesn't have form of material properties
Topology has entered the chat..