Photogrammetry is DEAD!
ฝัง
- เผยแพร่เมื่อ 7 พ.ย. 2022
- NeRF (neural radiance fields) is a promising new technology which has the potential to replace many of the current use cases for Photogrammetry. Here's a pretty basic discussion about what NeRF is, and how it can be used.
This is a very complex algorithm which requires a pretty good working-understanding on AI and rendering techniques to understand properly - so I did simplify a few things.
Here's a paper about NeRF - www.matthewtancik.com/nerf
Support this channel and get access to exclusive content on Patreon- / decoded
Facebook - / blenderdecoded
Twitter - / decodedvfx
#Blender #DECODED #B3d
Sorry about the crude geo-nodes point cloud demonstration. I only decided to make this video an hour ago, and I can't be bothered to make graphics for it. 🤷♂
No problem brother.. content matters🤗
all good
It's actually pretty good
I found the idea surprisingly cool and well done
@@golemtabak1183 thanks!
My experience has been that Nerf produces very squishy geometry that’s not always detailed. Looks good for video but not the best for creating detailed meshes. Maybe this is or will be changing.
Mesh generation is possible but the algorithms aren't tweaked for that use-case yet. Better meshing options will likely appear in the future.
If there’s some way to convert the neural render to clean geo with pbr it would be game changing
technically NeRFs don't produce geometry at all - you get a data structure that you can only query with an xyz coordinate and view angle, and it outputs an opacity and color - to view a NeRF you have to do that volume rendering process of sampling along rays and compositing the results by stacking those samples and compositing using the opacity values.
when NeRF demos show the ability to get a depth map from a scene, what they're doing is running that volume rendering process and just calculating depth to be the distance along the rays where the opacity starts being high - all the NeRF based tools that export meshes work using that same principle to estimate where the surfaces in the scene are, but the NeRF model itself has no internal representation of geometry. Directly rendering it *as* a NeRF without trying to convert it to a mesh is always going to look better - and its likely tools will start growing the ability to cut regions from NeRFs and composite them directly into scenes without converting them, which should even allow you to get decent results for glass (somewhat, NeRFs do model transparency well, but they do *not* model refraction or anything that would cause a ray to change direction - you can for example capture a good NeRF of a mirror, but it will interpret that as there being stuff *inside* or behind the mirror,)
NeRF models "learn" the scene by running that exact same volume rendering process *backwards* - rays are marched from the position of each pixel training image, the returned samples are composited and the resulting image is compared to the training image, then the model is updated in a way to make viewing the scene from that angle slightly closer to the training image - if you only have one image the resulting NeRF will only look correct from that exact angle because it has no way of knowing how far away the color information is, so the density would be evenly distributed along the sampled ray positions. As far as the system is concerned as long as things look correct at the position of the training images everything is fine. When you have multiple angles the possibilities are narrowed down significantly - the model has to put the information from each image at a position that doesn't *contradict* the other images - this explains a bit the sort of artifacts you get in regions of a NeRF there weren't many images of, they'll usually look perfect when viewed near the angles of your training images that saw those areas, but from any other direction you'll see clouds and streaks, as the image information kind of spreads to all places it could *possibly* be. That's, at least, the basic original NeRF formula from 2020 - there've been a lot of improvements on the concept since then, some of which do help coerce them to learn more physically plausible scenes that could get better results when trying to create meshes from one.
@@DECODEDVFX so what you mean is.. photogrammetry is not dead?
@@SamEmilio2 Photogrammetry still has 2 years left maybe. Thing is NERF works well with diffusion AIs, so its improving at a very, very rapid pace. In 2 years, even one photo of an object, is likely enough to generate a 3d mesh.
Photo -> Diffusion -> Multiple photos imagined from every angle (As long as the object is typical enough that the AI can understand) ->NERF ->3d mesh.
An important distinction between Photogrammetry and NeRF that I think some people miss is that NeRF (in its current state) creates the novel views of the subject that we see in the orbital renders on a per-pixel basis, so it eliminates any kind of mesh generation all together.
So from a workflow perspective, if you're looking for mesh generation for say a 3D scene asset, NeRF in its current form wouldn't generate any such asset, only a 'final' rendered output. That's not to say it wouldn't be able to synthesize a mesh in the near future I would bet, especially considering the depth/volume maps that are created in the NeRF process.
Yeah I've noticed a lot of people make that mistake too. I touched on volume rendering here, but I wish I stressed the fact it isn't a meshing tool more.
If you have the depth maps from various angles it should be quite easy to generate meshes from that. Or you can use the depth maps to do parallax occlusion mapping.
@@DECODEDVFX It totally threw me off at first too when I was digging into the method. I don't think it will be long though before meshes can be synthesized, especially with all the recent and what seem like lightspeed advances in AI generated content.
@@SchemingGoldberg Definitely!
@@HonorNecris meshing is possible with some implementations and it's compatible to photoscanning from a few years ago. But the current methods are mostly aimed at volume rendering. Like you said though, it won't be long before someone implements a much more robust remeshing system.
Complementary tech actually, NERF can enhance photogramettry by giving it more view point for generations, so the trade off of each other adds up to a better workflow.
Just subscribed to you. Please keep making videos! You explain things very clearly and concisely. You got a great voice too, very easy to listen to.
Thank you
Once I realized your mic changes colors, I was focused on that like a deer on headlights.
"Fuck yeah trigonometry or whatever" I'm dying 🤣 😂
Loved the video! Gotta readjust your mic tho! the text on your QuadCast should be directly facing you. In the video its pointing down to the bottom of your side desk.
When I record on camera I don't use a pop filter because it gets in the way, so I turn the mic slightly away to stop nasty plosive sounds.
1) Stable Diffusion to set an initial scene.
2) Nerf to extract geometry.
3) Outpainting to extend scenes into newly viewed areas.
4) Paint-by-words to extend to rear views of 3d objects via silhouette extraction and style guides.
5) Upsampling for LOD.
Viola! Infinite, high detail navigable 'levels' for any game with extremely low overhead and 'infinite' LOD.
I can even picture some ways to make the objects in the environment interactive and, with recent experiments into driving simulators with SD, it seems it's possible to simulate the physics as well!
I don't think stable diffusion can accurately recreate the same scene from multiple views. Can it?
@@DECODEDVFX Not as such, but you can use SD for the initial scene/pov, then let it hallucinate different povs with inpainting. Then save the new images as textures for the Nerf geometries.
Basically, get one pov, then while rotating around the object, use it's silhouette as a mask for inpainting, until you have the entire object 'painted' and save that.
Really, though, the idea is to use something like SD to 'imagine' the initial scene, then fill in the details as new areas become visible. I've seen some demonstrations with using SD to create a scene for a stationary pov which can rotate and it's impressive, though it doesn't remember what was in the scene before, so if you turn to the side and then turn back, it's all new imagery.
@@DECODEDVFX Understand, though, that this is all bleeding-edge stuff, so actual application and which models get used for which parts is very up in the air. The point is more that we're starting to develop the tools which are capable of fulfilling the roles you'd need for such a thing.
If you remember the game from the movie 'Her,' it's easy to point at some of the models which exist _now_ as early versions of that system. I think such systems are < 10 years away. We'll probably start seeing some generative stuff in the next wave of games, or the wave after.
Basically:
Stable Diffusion/etc...: Any scene from text.
DreamFusion/Magic3d: Text to 3d model.
Palm(er): Coherent voice interface and conversational model.
CLIP (and many others): Image to text. (model extraction)
Plus text-to-video and other models which are yet to be released. It's really just a matter of combining these with existing game design techniques and each other.
Whatever term we use (photogrammetry or NeRF), the holy grail for the visualization artist would be the ability to search for one or two photos of an object or scene on the internet, and automatically generate a detailed 3D model from it to use as an asset. If a human can recreate a 3D model from just an image, a computer should eventually be able to do the same.
I’m waiting for Midjourney to do the same.
I'm excited about the future:
Making 3D Models of all the Hot Supermodels Girls (nerd it out)
My Unity Assets gonna b sweet
@@dietrichdietrich7763 Don't want to be a party pooper but I'm just gonna point out that nobody is going to buy them if they can get simillar stuff for a lot cheaper using the same technology that mostly requires input and little to no technical know-how.
Insightful video. But most people don't understand the real purpose of Point Clouds. And I'm talking about professional ones, not some low grades made with an iPhone or Matterport device.
A high-end Point Cloud replicates a physical object (mostly a building or whole city) in the digital world with high measurement accuracy. It is the base for planning and simulation projects like construction, real estate design, film set production, and heritage building preservation.
Did anyone else have expect him to reveal that his room was NERF'd?
What was the source on the two scenes of the Office/Meeting Room and the Kitchen Stove? I don't see those on the NeRD project page.
www.matthewtancik.com/nerf
@@DECODEDVFX Cheers!
Great description! I do a fair bit of photogrammetry for museums and glossy objects are always a problem, as museum objects can’t be coated with powders or sprays to get good scans. So I’m really Looking forward to seeing how this technology progresses!
How did you get into this, if you don't mind me asking? I've always been intrigued about doing just this. Do you freelance, or work for a company that offers that as a service? Or do you work directly for a museum? Also just curious what your experience is with it.
@@xXWhatzUpH8erzXx i actually started photographing for some large museums almost 25 years ago and I still currently freelance for a few different museums. I also create VR and AR experiences for my own fine-art photography and I started capturing just a few photogrammetry models of sculptures for free for the museums about 6 or 7 years ago just to show them where technology was headed. (Hoping they would bite ;) It took several years before any of them started asking for 3d models, but now I get quite a few requests a year. I’ve also built some AR and Vr experiences for the national park service and a couple of museums…
Thanks for taking the time to explain this. I only just heard of this term and you did a great job of simplifying and explaining the whole thing.
Worth noting that the paper is two and a half years old and all we got so far is some more research on it, but no further breakthroughs or actual products in sight. Something useful might come out any day, but we might aswell wait years without any notable progress on it.
I don't mind this format as I'm always curious about new developments. I'd be really curious about a walkthrough of a NerF to photogrammetry model workflow. I think eventually photogrammetry software will use this to create even better models with realistic material properties - those clean depth maps could even be used to make height maps + normal maps giving us a low poly-high detail scan. It's only a matter of time 🤯
Couldn't agree more..
Thanks for the explanation. My brain could not comprehend how NERF differs from typical photogrammetry.
Yeah it's beyond confusing unless you have a really good grasp of light transportation rendering and AI neural networks. The people who write these papers don't exactly aim their work at laymen or artists, unfortunately. Long story short, photos go in, AI generated video comes out.
@@DECODEDVFX ai in same sense as other ai, used for old photo reconstructions, 3d generation just by text inputs etc? if so, then we got same problem as things, mentioned before: it need actually database of images etc, while i'm guessing it's provided by beta users etc? and this is never really 100% accurate, this is why i don't like with this kind of "ai", which is not really that. it is just averaging samples. classic fotogrammetry at least get 3d from from actual photos of the subject.
what u doin here
Great video! Thanks for sharing this topic, had no idea it was a thing. Appreciate you.
1:58 "FUCK YEAH! Trigonometry or whatever…" 😂
Put that on a shirt. I'll buy it.
"... hair..." 😄😄😄
I purposefully pointed to my beard 🧔♂️
This path is very promising. Taking into account amount of open source video data generated in last decade, there is a lot of data to train neural networks like this.
What "open sourc vide data" are you specifically talking about?
Two people have already gotten Nerfs into Blender. James Pearlman and CEB Studios. I plan to use Nerfs for virtual production when they come to Blender.
I just saw James' implementation on twitter. Very cool stuff.
What about Luma AI
Thanks for the info! I like the mix of formats. So far I've watched all the talking head vids, so I must like them! haha. Keep up the great content and thanks for sharing your knowledge.
Awesome, thank you!
BCON was filled with amazing people, including yourself! Amazing community. NeRF sounds nuts, def checking links
Very cool talking to you at BCON.
So is this new tech or technology coming to Blender? I'd be VERY interested in testing it and working with it in the future!
Hi, thanks you
Is it possible to do photogrammetry from only 4 images?
Not really. A few dozen photos at least are normally used.
@@DECODEDVFX Ok thanks you
It will probably not replace it. Since it does not actually capture the geometry it just represents it, so it might be wrong. In many usecases this is important. If you want to capture a crime scene you are going to take just a couple of pictures and the job is done. Same goes for engineering etc. For artists this is different they don´t care if things really work as shown in an animation like all those weapons in video games etc.
I want to be able to extract mesh and texture from it. Wonder if anything can do that with nerfs?
You can extract a mesh but it's not what NeRF is optimised for. Mesh extraction will improve over time.
Nice Ibanez back there - is that a 'Prestige" model?
Nah, nothing too fancy. I only just bought it after not playing since my teens. Basically starting from scratch.
Is there one available for use on a drone???
This is very good and I’ve subscribed. I do have a couple of questions if you have time to respond. First of all I really appreciate your style, not trying to act like some hyped up TH-cam video star. My partner laughed at your meeting Dick-heads comments. Anyway, just learning about Nerf. I used to do photogrammetry and dropped it a few years ago. So many problems. Now I have an opportunity with new Volume Walls for movies, tv series. Anyway, can this run on a modern Mac or do I have to go out and purchase a PC (yuk) so I can install NVIDIA GPU? I used to emulate pc software. Don’t know if that’s possible anymore. I would love to give this a try. Thanks
I haven't seen any implementations of it that run on Mac yet. I know Apple has done some research in radiance fields, but I haven't specifically seen anything about NeRF. I'm sure it'll happen eventually though.
Not related, but what photogrammetry software do you recommend?
Reality capture is the best paid app I've seen and it's fairly cheap. Mesh room is a good free alternative.
@@DECODEDVFX Thanks! I'll check them out.
U do realize that nerf needs to know every camer position. Because of that it uses the same tech as photogamaphy to get these locations
love your videos. Wish I could have gone to bcon!! I bet it was fun. I'm really curious how Blender will embrace AI. I like modeling, does this mean building up all those skills will not have been a waste of time?
No it isn't a waste of time. 3D modeling will still be a useful skill for a long time.
I wonder... is it the same attitude as the horseshoe smiths had when this "newfangled automobile" came along? They were arguing that it's just a toy that can break down and horses had been around since before history therefore it couldn't possibly be replaced as the main mode of personal transport?
@@FuriousImp Or is it more like autotune, which hasn't eliminated singers, but instead it empowered more people? All those photo filters on our phones, everyone is a photographer, but that hasn't hurt real photographers has it (or has it?). I was surprised that DJs became more popular after the invention of the CD player?! Evidently vinyl is still a thing. It's all very confusing to me. One thing I know though - it's really fun to use blender, and while I kitbash sometimes I still like to model from scratch. Today's AI may be like super advanced kitbashing?
@@powersprouter Well there is an interesting moral discussion to be had for sure. For instance I consider autotune to be cheating in the same way an athlete might take steroids or blood doping. It is an enhancements of your skills that you haven't put blood sweat and tears into, so it is very unfair and unethical towards those who play by the rules.
Where can we try this?
Nvidia instant NeRF is online but it needs to be compiled. Luma AI has a form you can submit to be a tester.
with all these Nerf tech, do we still need modellers in the future?
Since we can just scan eveything, Unless we are doing abstract stuff.
Modelers will have a place for the foreseeable future.
Another amazing video brother! keep it up
It was a class about photogrammetry and NeRF. congrats, I enjoyed it a lot.
If NeRF requires multiple image frames and the angle of the camera (according to Nvidia's own information on it), it's still technically photogrammetry, is it not?
No. Photogrammetry is 3D mesh data generated from photos. NeRF is a synthetic neural output from photos. There isn't necessarily an output mesh with NeRF. It outputs a volume render.
@@DECODEDVFX 3D mesh generation from photos is just one form of photogrammetry. Photogrammetry means measuring with photography. NeRF is technique that could be used for photogrammetric applications.
From what I've seen you don't have problems with reflective surfaces with NERFs like photogrammetry does.
Great video Awesome guitar you look like you could shred!
It's actually brand new. I haven't really played for about ten years.
I tried it yesterday
the quality of reconstructed output is not that great
it might improve in future
360 captured panorama will give better results
In the same way we have thousands of jpegs in our laptops & phones, we will have thousands of Nerfs someday sooner than later. I’m just starting to imagine the impact.
I enjoy those video about general topic !
Thanks!
You pointed at your head, when you said "hair"
I pointed to my beard
Cool guitar! What are you playing on that thing?
Nothing at the moment. The little practice amp I bought for it is broken
This kind of stuff, combined with the AIs that can create things from text strings, and presumably eventually voice prompts, VR, and things like Unreal Engines Nanyte, we are literally only a few years away (if that) from a vr version of a holodeck.
"computer, give me 1970s Brooklyn, I'd like my avatar to be a hippy" etc.
Assuming we're all still alive in a year, what a time to exist in. Truly incredible stuff, tech is.
I'll be straight in that Irish pub from voyager
volume rendering been around since the 70s?
Yes, but it was impractical on the hardware of the time so other rendering techniques like rasterization and retracing were preferred.
Can Nerf be used with unreal engine 5 to assist with building virtual environments?!
How well would it work with faces?
Really well. The temporal versions can even replicate moving faces
Hmm...that sounds ok, but you'll still need a virtualized photogrammetry-like data processing pass to get the actual 3d data like an obj file, right?
What I've seen so far is "just" interpolated depth data with captured texture slapped onto it...
And then you need another texture pass where another neural network analyzes the texture data and figures out reflectiveness and transparency. Which to me seems way harder than generating a point-cloud from a matrix of depth textures.
NeRF is the technology but what software is out there leading the way at this point in time to present a real time development phase to its expected maturity?
Are we going back to sprite-based gaming, where it chooses a prerendered angle to display like in older Mario Kart games?
I hope so
I do understand the photogrammetry math and know my way around deep learning professionally. NeRF is a blessing, but it has one drawback; it does not provide absolute scale, whereas photogrammetry does as long as you specify the pixel pitch correctly.
However, absolute scale is only required for measurement and not shape reconstruction, so using NeRF for object recreation is totally fine.
PS. You can easily import a NN in Python and run it in Blender. Someone will certainly write a script to do this.
Based on a few papers I've read people seem to be working on a solution for absolute scale. But I don't know much about how that stuff works. It seems like a lot of research is being done to build on top of the basic NeRF method.
You can run NeRF in Blender wtf really?
@@openroomxyz theoretically yes, someone could make an addon to let you use nerfs in blender.
Awesome more stuff about this please! 😎😎😎👌
Photogrammetry/photoscanning isn't just used for 3D, it's also used for 2D. At its core, though, it's used for generating meshes (or, at the very least, point clouds) so they can be used as a base/source, used for all kinds of things - physics sims (collision), rendering, modelling/rigging/animation, etc
Saying photogrammetry is dead because _one specific niche usecase of it is slightly better with this AI tool_ is... pretty clickbaity actually.
Was really excited to see what Hot New Thing had replaced photogrammetry's primary use, mesh generation, well enough to actually earn the title of this video, but alas, there's still nothing. But hey, you got a view and a comment and "engagement" on the like/dislike buttons, so congrats. Your clickbait worked.
Traditional photogrammetry will have a place for a long time yet for applications that need accurate meshes. Medical science, forensics, etc. But it's role in 3D art (which is what this channel is about) will be seriously diminished in many ways. The really exciting possibility is a merger of the two technologies. Imagine accurate photogrammetry meshes with reflection maps and materials generated by NeRF!
So volume rendering as you describe isn't as far back as you'd think, I'd say Terminator 2 is the CLOSEST to a volume, but it was still a forward render....
Point clouds and splats and all that really came up in the 90's and that shit is super hard. For example notice there is almost 0 volume rendering in web based platforms. They can do it, but it's so heavy it's not worth it except for like medical scans..
Volume rendering in general is pretty old, but you're are right Raycasting through volumes is more recent.
You're briefly famous at the Blender Conference the same way I'm briefly famous at planetarium conferences. I say that's the perfect amount of famousness. Keep up the good work, sir!
The end of 3D modeling... final nail. Excellent video though.👏
I really really love videos like this!
It looks like you forgot to turn on denoising when you rendered your jumper pass.
You mean the creases? Yeah, it's thing lighting. One super-strong directional light makes all my clothes look creased to bits.
@@DECODEDVFX no, it actually is an incredible amount of noise. Much like rendering in Cycles with denoising off. I'm watching on an iPhone so perhaps the way HDR works with TH-cam is emphasising something barely visible on other screens. It does seem screwy sometimes.
@@LongPeter it's just recorded on a webcam so it's a bit noisy. TH-cam compression doesn't help either.
Photogrammetry isn’t dead though? Maybe it’s just… aging?
great explanation man.
Suggestion to anyone watching this, try this with stable diffusion and images from outpainting
Just add all the maps together
They just need to optimize It for generating a mesh, right now this technique works better for video or 3d image generation, not meshes
Better mesh solutions will appear over time.
I'd like to ask you this question: Do you think 3D modelling as a career will eventually die because of advances in AI? Will those seeking employment in the 3D marketplace be forced to diversify ?
3D modeling will have a place in the pipeline for the foreseeable future.
Any technology thats cuts down on man power and improves efficiency will always win out in the long run over tech that doesn't. Thats why nerf is the future over photogrammetry, I can't wait to see entire cities generated in detail from just Google images alone with this technology at far better creative speeds
Very good indeed!! Thank you!!
Photogrammetry vs Nerf? Use case dependent... maybe it's safe to say that for now nerf is good for rendering/visuals/vfx, but is awkward when precision is needed (3d printing, high poly mesh). And Nerf still needs the first stage of photogrammetry - solving for camera positions ... generating point cloud... From what I've seen, I find its strength is in the ability to capture material&lighting properties... transparent/glass/metal and delicate meshes, hair, plants, grass(?)...
Awesome, thank you!
This shit's nuts!
Fascinating stuff
It is good but Photogrammetry is going to stay , Nerfs use imagery, hence Photogrammetry. Nerfs are cubes instead triangles
The people coding this stuff must be geniuses.
two images that fall on our retinas are two two-dimensional images, but the world is still three-dimensional, four, etc. if we want something more than a simple stereoscopic image, for example to interact with 3d holograms in a room, we need 3d objects, which does not mean that ai will not make them. but please don't kill my photogrammetry yet. the essence is to combine.
I think i have the same Ibanez as you do.
Mine is the GRG121DX. I just bought it.
@@DECODEDVFX probably better then mine then. Bought mine like 4 years ago.
i have decoded that you are a shy person . I hope this comes soon
Way cool!
Oh I know I use photogrammetry for 3d modeling.
Did anyone else click this link *clutching their papers* to see NeRF out in the wild? 😆
ai getting insane
I thought "Modeling Is Dead", LOL... 😂😂😂
Fuck yeah triganometry or whatever 🤣🤣 that was too funny 🤣
This is loco, esé
Look at Jon Barron - Understanding and Extending Neural Radiance Fields video, HfJpQCBTqZs
love u man
Photogrammetry has been around for about 100 years and is far from being dead...
Noicee
So far from actually replacing photogrammetry.
fuck yeah, trigonometry!
lmao. Love it.
This is so awesome! Lets go 3D Ai
Dead? Show me one commercially-available product that can use NERF
sick. Im to poor for a new iphone.
Fuk yeah, trigonometry.
Dreamk on
not for commercial applications, like crime scene investigations or land topo survey, NOT DEAD just hate direct comments when not looking at the bigger picture. TH-camRS ARE TH-camRS
Not so fast. Proof? Megascans in unreal engine. It will be in the future.
Photogrammetry will be here for 5+ years
current NERF isn't good enough .
I'm interested to see how it'll develop. Lots of breakaway projects based on the original NeRF papers are being developed.
@@DECODEDVFX yes, hopefully i'm wrong about the 5+ years. ☺️would love to have it soon.