Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields

jonbarron.info/mipnerf360/
Though neural radiance fields (NeRF) have demonstrated impressive view synthesis results on objects and small bounded regions of space, they struggle on "unbounded" scenes, where the camera may point in any direction and content may exist at any distance. In this setting, existing NeRF-like models often produce blurry or low-resolution renderings (due to the unbalanced detail and scale of nearby and distant objects), are slow to train, and may exhibit artifacts due to the inherent ambiguity of the task of reconstructing a large scene from a small set of images. We present an extension of mip-NeRF (a NeRF variant that addresses sampling and aliasing) that uses a non-linear scene parameterization, online distillation, and a novel distortion-based regularizer to overcome the challenges presented by unbounded scenes. Our model, which we dub "mip-NeRF 360" as we target scenes in which the camera rotates 360 degrees around a point, reduces mean-squared error by 54% compared to mip-NeRF, and is able to produce realistic synthesized views and detailed depth maps for highly intricate, unbounded real-world scenes.

[@cailihpocisuM]

Amazing results, truly compelling!

[@ThetaPhiPsi]

The depth map is amazing, wow!

[@legoworks-cg5hk]

What a time to be alive

[@BAYqg]

dear fellow scholars

[@Halcyox]

Quite excellent work that is done here!

[@HaroldR]

This is great!

[@exhibitscotland360]

Very Interesting.

[@superatomic9761]

this is amazing. does it work with images of people?

[@sinlife484]

amazing! I'm curious if it works for indoor scene reconstruction, could you please tell me ?

[@Neptutron]

This is amazing! Given the precision of the depth maps is greater than the input of SVS, could this be used for more accurate photogrammetry?

[@legoworks-cg5hk]

Exactly what I was wondering

@@legoworks-cg5hk hopefully RealityCapture or Agisoft adopts this fast

[@legoworks-cg5hk]

@ is there a way to use reality capture without nvidia?

@@legoworks-cg5hk sadly no.... Agisoft can be used with any GPU, but is slow as hell without CUDA... one solution is cloud processing

[@legoworks-cg5hk]

@ only problem with agisoft is that you have to pay for it to export models

[@bharatsingh430]

Looks pretty amazing! To import this to a graphics engine and be able to render new objects in this scene, we would also need the light-sources, material properties (diffuse/specular etc.) and normals (unless the depth maps are super accurate) - wonder if those could also be recovered by modifying this technique..

[@timsousa3860]

Seems quite the challenge

[@shubhashish7090]

can we exatrct the mesh from this to be used in any traditional game engine

[@willhart2188]

I'd like to try this in VR.

[@user-jl7xw9rc1g]

Genius , abusolutely.

[@khaledbouzaiene3959]

can we can extract 3d model maps ?

[@changgongzhang6641]

For the regularizer at 6:40, the minimum of Loss_dist is achieved by setting w(u) = 0 everywhere, right? wondering how it can become to a delta function?

[@jon_barron]

You are correct, the reason that w(u) doesn't get set to zero everywhere is because that would cause the data term of the loss to be extremely high. In that animation I normalized w to sum to 1, which in practice is what happens during training because of the data term.

[@changgongzhang6641]

@@jon_barron Thanks a lot for your explanation!

[@GmZorZ]

quite an interesting way of representing depth, how is it done? it’s about time black and white maps become a thing of the past!

[@jon_barron]

This is the "turbo" color map: ai.googleblog.com/2019/08/turbo-improved-rainbow-colormap-for.html

[@GmZorZ]

thank you so much! very informal, i’d have guessed more bit variation in an image presents much more detail, for me it’s really all about what you can fit in standardized 32bit formats! I will incorporate this in my own projects to further normalize the standard.

[@marknadal9622]

​@@jon_barron is color/density at depth position determined by the training? In all the videos it seem to be implied that its known as input, but that clearly isn't possible from a 2D image without prior object size training. Sorry for dumb Q.

[@kwea123]

@@jon_barron Thanks for the great article! I always used jet, and was wondering how turbo is different. Now turbo looks better to me!

[@pretzelboi64]

If you don't even know what color mapping is, I don't think you're qualified to talk about what should be things of the past lol

[@user-sc5xk6vy8j]

I don't understand why distortion loss can be written like this. Doesn't the cubed term appear?? Is there anyone who knows??

[@user-by6fr4dj4k]

which lab did you cooperate in Harvard?

[@Askejm]

will it be publically availabe/is someone working on an implementation?

[@jon_barron]

Yes, we'll be releasing code soon.

[@Askejm]

@@jon_barron does it run on windows and will it have a pretrained model?

[@saltygamer8435]

have you released the code?@@jon_barron

[@Instant_Nerf]

Is video playback possible wirh nerf?

[@jon_barron]

Yeah check out th-cam.com/video/zBSH-k9GbV4/w-d-xo.html

[@HA-cy4vx]

wowwwwwww

[@mahmoodmasarwa3374]

any way we can test it ?

[@jon_barron]

Sure! github.com/google-research/multinerf

[@user-hi2bz8ds2h]

Hi Jon, I saw the newest project provided by you and your colleagues, Ben Mildenhall. I want to say that it is so impressive and simultaneously useful for plenty of new ideas and projects. We are working on some interesting projects where we think Anti-Aliased Grid-Based Neural Radiance would be the best option to increase the effectiveness and productiveness. Hence, can we have some direct conversation about the project ? I am really looking forward to hearing from you.

[@ncmasters]

Is the code available yet?

[@jon_barron]

Yep, here you go: github.com/google-research/multinerf

[@ncmasters]

@@jon_barron Thanks

[@Padoinky]

Have to goggle this to even understand what it is about