GTC 23 Nerfstudio: A Modular Framework for Neural Radiance Field Development

Doc: nerf.studio
Neural radiance fields (NeRFs) are rapidly gaining popularity for their ability to create photorealistic 3D reconstructions in real-world settings, with recent advances driving interest from a wide variety of disciplines in academia and industry. However, due to the flux of papers, consolidating code has been a challenge, and few tools exist to easily run NeRFs on user-collected data. I'll introduce nerfstudio, an open-source Python framework we recently released to address these issues by consolidating NeRF research innovations and making NeRFs easier to use in real-world applications. I'll discuss its development and recent updates such as visual effects integrations and more.
Bio: Angjoo Kanazawa (people.eecs.be...) is an Assistant Professor in the Department of Electrical Engineering and Computer Science at the University of California at Berkeley. Her research is at the intersection of Computer Vision, Computer Graphics, and Machine Learning, focusing on the visual perception of the dynamic 3D world behind everyday photographs and video. Previously, she was a research scientist at Google NYC, and prior to that she was a BAIR postdoc at UC Berkeley. She completed her PhD in Computer Science at the University of Maryland, College Park, where she also spent time at the Max Planck Institute for Intelligent Systems. She has been named a Rising Star in EECS and is a recipient of Anita Borg Memorial Scholarship, Best Paper Award in Eurographics 2016, Google Research Scholar Award 2021, Spark Fellow 2022, and Sloan Fellow 2023. She also serves on the advisory board of Wonder Dynamics, whose goal is to utilize AI technologies to make VFX effects more accessible for indie filmmakers, and PainScan, whose goal is to make chronic pain visible in 3D so they can be properly diagnosed and treated.

[@zzzzzzz8473]

fantastic overview , really exciting developments in NeRF , thanks !

[@OlliHuttunen78]

Great presentation! I've certainly come across NeRF studio while researching this topic, but it's really great to hear what's behind it and who does it. Good luck with your project. We live in exciting times.

[@dialectricStudios]

legends, all of you. I could see a future where there's a NeRF renderer built into blender... we're not there yet but the integration between the two apps right now is so awesome!!

[@mooncop]

based! cannot wait for these to be used for grounding foundation models in hyperreality 👁️‍🗨️

[@szczyglo74]

Amazing technology. Thank you. Maybe someday it will work in real time. I mean training based on prompts or other data. We will let the algorithms dream and we will have the ability to jump into their visions. It will be more than modern computer games or vr movies.

[@jimj2683]

The next step would be to use sensors to capture the physical interactions of objects. That could give the model a physics model of the real world and allow it to be used in games.

[@zenbauhaus1345]

great stuff!

[@jimj2683]

When will Google Street View be made in 3d? I want to drive around and explore the world as if it is a game or simulator.

[@shania5214]

Thanks for this informative talk! Is there perhaps a tutorial on how to use the webviewer?

[@grahamwebster8913]

The only downside to a web viewer is it’s not gonna fly in the DOD or government. Altho from my last look you can run it on a VM used as a sort of server which IT is a bit more okay with.
Would be nice to have just a straight up local desktop viewer

[@naninano8813]

nerfstudio turned out to be suitable for gaussian splats too

[@g-program-it]

cool

[@li_tsz_fung]

Hello, what do you think about the future of NeRF? I don't know anything about NeRF development, but I've seen a lot of related demos.
I feel like there is a big difference between NeRF based on photos, and generated nerf.
Performance-wise, it seems like NeRF can one day become a fully interactive environment. But we wouldn't be just creating interactions solely based on real life scenes and objects.
I thought if Google and Apple reconstruct all their street view and Google earth source photos to NeRF (their new map release seems having), they can then train a network on top of that, and use it to generate scenes. But that would still be mostly buildings and cars.

[@AikoBasicHula]

英語はよくわからないですが、聞いてるだけで、ワクワク🎉します。❤❤❤❤