I think I've just found the solution for an ambitious Unreal 5 cinematic project, that was, until about 10 minutes ago impossible for one man to produce. Thank you for the insights.
@EveryPoint Is this good enough to export a series of frames that could be used for photogrammetry? As if you didn't get enough data on site for whatever reason, and you could use this to fill in the blanks, so to speak?
Unsure if that’s possible. We would need to test that theory. However, Nvidia do have nvdiffrec that neurally renders geometry and PBR textures. That may be the answer
So I work in film and VFX, would this method work for creating a 3D render of a city to be used in a production lets say for use in "Unreal Engine or another 3D program?
@@EveryPoint Adobe Substance has the python API and nodes to allow an AI to generate 3d assets and textures. This is what I'm researching for this software which led me here....
That is an option. We find that matching the output video resolution to the source imagery works well. However, overall most NeRF output is a little "fuzzier" than real life.
Seems like nerfs could really benefit from transfer learning. On 2d object detection you can train a decent model with 100 images with good pretrained weights. I guess with common scenes sky/tree/house patterns can easily be learned
Hi @EveryPoint. Its still not 100% clear if nerf is ready for commercials. How to start with own 3d games/walking experience. May i ask you to make some video/post explaining which instruments are free to use/commercials ready and which are not.
Should it? Or should software and hardware systems catch up to this technology where the Gltf format can't replicate what a NeRF can visualize? As Matt's first statement said: NeRFs are not meshes.
Great question. Stereo pair cameras can aid in providing depth and positional data. IMUs and other technology can be leveraged as well to provide a good SLAM based camera pose output for training NeRFs.
Therefore, both photogrammetry and NERF need a set of images with known posas input. However, modern photogrammetry technology does not require high accuracy of pos, and even does not need posas input at all. What I want to ask is, does NERF require high accuracy of input attitude?
I am not sure why AI researchers seem to ignore the fundamental advantage of our species. Our ability to understand & construct complex from simple. We are trained from babies to understand basic rules for everything we see and work out what & how for everything from shapes in the cot onwards. If I dont understand something I break it down until the rules kick in. If a Nerf or meshing algorithm just applies brute force it will never be really efficient, but if it understands what a car, column or crow is it surely it will then be able to adapt a 3D library element to its surface\field far quicker in much the same way the 2d AI image generators are doing.
there are other videos which are shorter and more general, but i was looking exactly for this discussion and exploration. youtube is a vast landscape :) something for everyone
I think I've just found the solution for an ambitious Unreal 5 cinematic project, that was, until about 10 minutes ago impossible for one man to produce.
Thank you for the insights.
@EveryPoint Is this good enough to export a series of frames that could be used for photogrammetry? As if you didn't get enough data on site for whatever reason, and you could use this to fill in the blanks, so to speak?
Unsure if that’s possible. We would need to test that theory. However, Nvidia do have nvdiffrec that neurally renders geometry and PBR textures. That may be the answer
So I work in film and VFX, would this method work for creating a 3D render of a city to be used in a production lets say for use in "Unreal Engine or another 3D program?
Eventually NeRFs could be used in this manner. The tools to interact between NeRFs and game engines/animation software is yet to be developed.
@@EveryPoint Adobe Substance has the python API and nodes to allow an AI to generate 3d assets and textures. This is what I'm researching for this software which led me here....
Also Blenders Geometry Nodes could be used
Luma AI now has a UE5 plugin. Excellent video, btw! Thanks!
Will nerf produce a video of the same resolution the images were in?
That is an option. We find that matching the output video resolution to the source imagery works well. However, overall most NeRF output is a little "fuzzier" than real life.
Started also with the TI 59 :)
Seems like nerfs could really benefit from transfer learning. On 2d object detection you can train a decent model with 100 images with good pretrained weights. I guess with common scenes sky/tree/house patterns can easily be learned
There is a large opportunity for optimization. We imagine different NeRF methods could be optimized for different use cases.
Hi @EveryPoint. Its still not 100% clear if nerf is ready for commercials. How to start with own 3d games/walking experience. May i ask you to make some video/post explaining which instruments are free to use/commercials ready and which are not.
Export Nerf as Gtlf. That should be the goal.
Should it? Or should software and hardware systems catch up to this technology where the Gltf format can't replicate what a NeRF can visualize? As Matt's first statement said: NeRFs are not meshes.
@@EveryPoint that's a harder but good alternative I can say... I bet my money on the first one :)
Thank you for this great interview!!
what's the discord server?
Here is an invite link: discord.gg/yNMkx8Xx
@@EveryPoint this link has expired..?
Is HDR nerf possible?
Technically, possible since HDR just has more precision per color channel.
How close NeRF to SFS (Shape from Shade)? thank you
not close at all
Nerf studio for pytorch MPS please!
Jump on the Nerfstudio Discord and ask about it!
I don't think it would replace photogrammetry for commercial inspection purposes.
We do not think so either.
As soon as the word nueral, ala..think, guess, interpret, interpolates, etc is introduced, its no longer real world data.
Hi, thanks you
Is it possible to do photogrammetry from only 4 images?
Yes, but your results would be limited.
@@EveryPoint Ok thanks, with which software can I use only 4 photos please? because with RealityCapture and Agisoft Metashape it does not work
For clarification, no software can handle 4 random photos. They would need to be properly spaced with known camera information.
@@welsonfy5246 I once did a test with 2 pics (Colmap +OpenMVS). Worked surprisingly well.
@@pauljojo1318 Thank you for your answer, do you have a tutorial please?
I need to export the 3D model from a nerf photogrametry
dont bother, they are terrible
@@TheShaunlamont Maybe Luma AI could be good?
Dual cameras, like binocular, would be better than single view for on the fly depth scaling? We do it all day.
Great question. Stereo pair cameras can aid in providing depth and positional data. IMUs and other technology can be leveraged as well to provide a good SLAM based camera pose output for training NeRFs.
Greatt
Therefore, both photogrammetry and NERF need a set of images with known posas input. However, modern photogrammetry technology does not require high accuracy of pos, and even does not need posas input at all. What I want to ask is, does NERF require high accuracy of input attitude?
?
photogrammetry and nerf both use colmap least squares type camera target translations..and photog needs much higher camera pose info than nerf does
Finaly, minecraft 2
I went from Welder to blender.. and it went all down the rabbit hole.
Haha!
I am not sure why AI researchers seem to ignore the fundamental advantage of our species. Our ability to understand & construct complex from simple. We are trained from babies to understand basic rules for everything we see and work out what & how for everything from shapes in the cot onwards. If I dont understand something I break it down until the rules kick in. If a Nerf or meshing algorithm just applies brute force it will never be really efficient, but if it understands what a car, column or crow is it surely it will then be able to adapt a 3D library element to its surface\field far quicker in much the same way the 2d AI image generators are doing.
Semantic understanding is the way!
Qualitative and quantitative are diametrically opposed values...seeing is not measuring
No
Too long winded.
Not according to the viewer retention metrics ¯\_(ツ)_/¯
there are other videos which are shorter and more general, but i was looking exactly for this discussion and exploration. youtube is a vast landscape :) something for everyone
Best minutes?