Making a NeRF movie with NVIDIA's Instant NGP
ฝัง
- เผยแพร่เมื่อ 31 ก.ค. 2022
- NOTE! I made a newer video for the updated instant-ngp software: • Updated: Making a NeRF...
These notes include the command lines you'll need.
Search on "Dyanna Skunk Somerville" to find more information about the robot sculpture captured. See rafiproperties.com/news-and-u... as a start.
Instant NGP software and build instructions: github.com/NVlabs/instant-ngp
Problems? For help, see the Discussion area on the site github.com/NVlabs/instant-ngp... and search on all issues at github.com/NVlabs/instant-ngp...
Complementary blog post and video with more (slightly dated) info: developer.nvidia.com/blog/get...
Need more help? Try the NeRF Discord server: / discord
The commands I used in the conda command window ("ngp" is an environment I earlier created for instant-ngp; see below):
cd C:\Users\ehaines\Documents\_documents\Github\instant-ngp
conda activate ngp
cd data/nerf/robot
(COLMAP must be installed and add COLMAP-3.7-windows-cuda to the $PATH.)
(OPTIONAL: if you took a video instead of a set of images, do this next command to pull images out of your video and put them in the "images" subdirectory:)
python C:\Users\ehaines\Documents\_documents\Github\instant-ngp\scripts\colmap2nerf.py --video_in YOUR_VIDEO.MOV --video_fps 2 --run_colmap --aabb_scale 16
(Perform feature matching among your set of images:)
python C:\Users\ehaines\Documents\_documents\Github\instant-ngp\scripts\colmap2nerf.py --colmap_matcher exhaustive --run_colmap --aabb_scale 16
(Sometimes COLMAP will fail, in which case you need to take more photos.)
(Generate NeRF and set camera path:)
cd C:\Users\ehaines\Documents\_documents\Github\instant-ngp
.\build\testbed --scene data/nerf/robot
(FFMPEG must be installed first before generating a video - test first!)
(Save camera path and NeRF solution:)
python scripts/run.py --scene data/nerf/robot --load_snapshot data/nerf/robot/base.msgpack --video_n_seconds 8 --video_fps 60 --width 1920 --height 1080 --mode nerf --video_camera_path data/nerf/robot/base_cam.json
(Generates video.mp4.)
Initial installation notes:
You need to install COLMAP (and Anaconda, if you want to control Python that way). Search the internet; you'll easily find them. You may need to add "...\COLMAP-3.7-windows-cuda" to your PATH to get colmap2nerf.py to run.
The anaconda environment setup commands I personally use to install the packages needed (just need to do these once for the project, other than the "activate"):
conda create -n ngp python=3.10
conda activate ngp
cd C:\Users\ehaines\Documents\_documents\Github\instant-ngp
pip install -r requirements.txt
conda install -c conda-forge ffmpeg
The above creates a conda "environment" called ngp, makes it the active one, then installs the software you need. When an environment is active, you can "conda install" from anywhere.
Also, if you run into "ModuleNotFoundError: No module named 'pyngp'" when running run.py, first run in your conda window:
python -V
to see the version number, e.g., "Python 3.10.4". Then, change CMakeLists.txt in your topmost directory instant-ngp from:
find_package(Python 3.7 COMPONENTS Interpreter Development)
to
find_package(Python 3.10 EXACT COMPONENTS Interpreter Development)
- note you don't need to put the ".4" in "3.10.4". Delete the "build" directory and remake everything. That should solve it.
One more detail: you won't see the DLSS option if you don't have an RTX card.
![NVIDIA Just Made AI Photogrammetry 1,000x Faster [Instant-NGP]](http://i.ytimg.com/vi/fvXOjV7EHbk/mqdefault.jpg)
![[4K] TREASURE(트레저) “KING KONG” Band LIVE Concert 킹콩은 라이브를 찢어🦍 [it’s KPOP LIVE 잇츠라이브]](http://i.ytimg.com/vi/p8bLLOxPDD8/mqdefault.jpg)
Hi, may I ask how long did it take you to finish colmap2nerf (extraction /matching)?
Currently I'm using 3080ti and colmap 3.8 on ubuntu 18.04, however, even the training is fast, it takes me over 30minutes to convert
Right, anything involving COLMAP (a free set of tools, not made by NVIDIA) is not accelerated by GPU, AFAIK. And that correlation process can take a while, for sure. I don't have tons of experience with instant-NGP (believe it or not), but what I tend to do is trim down the training set to 50 pictures or less. I think more than that is overkill. Usually pretty easy to trim back, as a video extraction gives lots of nearly the same or blurry images. Anyway, less images means COLMAP's view correlation goes a lot faster, from my experience. For example, I just correlated 54 images on my 4-year-old CPU in 4 minutes. I think that the time goes up by something like the square of the number of images, so 150 images might well be 9 times as long as 54 - 30+ minutes.
@@MrErichaines That really helps, thanks a lot for the explanation!
Thanks for sharing, I wonder how many images do we need? What’s the minimum?
Sorry, I just noticed this comment. There are some systems that can take as few as three images and interpolate from those. I'm not sure the minimum for instant-ngp. I tend to take 25 or so, and easier still is to just take a video. Anyway, experiment and find out! The system resolves in just a minute or two, so it's easy to try things out.
When trying to render the video I have tried what you mention but I keep getting this error... any other ideas?
Traceback (most recent call last):
File "scripts/render.py", line 16, in
import pyngp as ngp # noqa
ModuleNotFoundError: No module named 'pyngp'
I have tried with different versions of pyhton, changing the CMakeLists.txt but there is no difference...
Thanks in advance
I ran into this same problem, and I explain exactly what I did to solve it in the notes below the video (click on "...more"). You need to make a small local change to CMakeLists.txt and make things again (me, to be safe, I delete the "build" directory to force a full rebuild). If you're still stuck, let me know.
i got that error too..
the pyngp module generated by cmake was for py ver 3.10
even though i was running py ver 3.9!
i ended up cloning the environment (so no damage was actual done) and upgraded py's ver to 3.10
then reinstalled some of the packages to make sure they were working correctly (i remember numpy and pillow being some of them)
a super simple solution is to run pip install 'moduleX' (no apostrophes) in the cmd
I ran into this with a number of modules that seemed to not be installed, despite having just done a clean install of 3.11
so anytime you see the error : No module named 'moduleX', just run pip install 'moduleX'
I'm having issues training on my own images with colmap. I can get the fox loading fine because it has the transforms.json. When I try to train on my own data it doesn't seem to use cuda with colmap automatically, gpu utilization is 0, cpu is 100%. The colmap standalone managed to generate a point cloud with the ini file but no json???
So when I use the command line it starts to calculate but it takes 12 hours to get anywhere... and I'm scared for the processor.
Pretty sure the colmap path is correct in the environment variables. Is there a flag to force it to use cuda/gpu at that step?
I don't know much about COLMAP but I think you're right, there's some GPU mode for it. One place to get help is on the Discord NeRF server: discord.gg/2nEDC52hs8
@@MrErichaines cheers, woops wrong account but yes, it turns out that I was actually using the GPU. Check with gpuz and it shows utilization. Just seeing if I can render a video path now without Melting my house down
Can we use 760 image that each one has 1980x1920 size? I got an error when i try to attempt this.
Do you mean use 760 different images? I can imagine running out of memory. The website page github.com/NVlabs/instant-ngp/blob/master/docs/nerf_dataset_tips.md says "It is recommended to choose a frame rate that leads to around 50-150 images." In the notes for my video you'll find links to discussion forums, etc., if you want to ask - I know only a bit. Also, this video is dated - my newer one's at th-cam.com/video/3TWxO1PftMc/w-d-xo.html
When you ran this it you got a warning that it used transformers.json rather than base. Is that supposed to happen?
Sorry, I just noticed this comment. I'm not sure where in the video you mean, from my quick skim through. But, whatever the case, I just made an updated video with more tips, since instant-ngp's had a number of improvements made to it. It's at th-cam.com/video/3TWxO1PftMc/w-d-xo.html
When I try the final step to render the video, I get CUDA_ERROR_OUT_OF_MEMORY (I have an rtx 3060 with 6gb), what can I do to reduce the memory usage ?
I'll poke around and see what I can find out. This (somewhat dated) video might have something about it, in the meantime: th-cam.com/video/z3-fjYzd0BA/w-d-xo.html - I feel like he talks about running out of memory, but I forget.
The advice I've learned is to go to the Github repo's issues section and search all issues (not just those that are open) for solutions. In your case, issue #576 there seems to apply: downscale the resolution of your input. Posting in the Issues any problems (and especially solutions) is a help to all users.
i usually either downscale my images or take some out so that i don't have too many
Hi, when I’m looking for the dlss option, I can’t find it, and what I have in it’s place is dynamic resolution… is this because I’m using a gtx card instead of rtx?
Huh, I wonder if the interface has changed - I haven't updated in awhile. I'll try to take a look. I show the option's location around 5:50.
DLSS is exclusive to RTX cards, so your assumption that it's because you have a GTX is correct.
@@hherpdderp Thanks! I couldn't figure this one out. I've made a note now at the very bottom of the description text.
@@MrErichaines sorry if I sounded sarcastic about that, I wasn't. I'm sure like many things Nvidia they just chose not to allow the software to do it on older cards .they should really should just give users a choice.
@@hherpdderp I didn't think you were at all, FWIW! From what I know about DLSS (I work for NVIDIA), you truly need an RTX card to make it work - it needs the special Tensor Cores hardware for AI to speed up evaluation of the neural net performing the process. The specialized hardware make a huge difference in speed - it's not marketing hype.
What if I already have my images and COLMAP cameras.bin files? How do I get the transform.json?
I'm afraid I don't know, I've always just used COLMAP to get this file. You might ask on the support forums I mention in my notes for this video, above.
curious if you have any insight why instant-ngp seems to only work well with image data that's in a 360 orbit around a focal point, vs say the Luma app on iOS that seems to do just fine with images shot on a linear fashion (th-cam.com/video/YX5AoaWrowY/w-d-xo.html) the guys at digital corridor seem to have no trouble with this
Well, Instant-NGP is free research-level software, research focused on speeding things up (days to minutes). There's other research focused on improving quality. You might want to give Nerfstudio a look. It has 5 training models currently, Instant-NGP is just one of them.
Any chance anyone has a script, or knows of one, that can carry out this process in a loop? Like a Python script to process batches of photos into datasets and then into NeRFs ?
Sorry, I just noticed your comment. FWIW, I made an updated video about instant-ngp yesterday, showing how instant-ngp has gotten easier to use and more batching-friendly: th-cam.com/video/3TWxO1PftMc/w-d-xo.html
What’s the real world use case for NeRF’s?
Karen x or Karenxcheng
We'll see! In the past two years we've seen hundreds of papers published in this area. Not being sure myself, at this point I'll go with Franklin's famous quote, "What is the use of a newborn child?"
@@MrErichaines mesh generation using this would be neat! though working with volumes seems quite the challenge..
@@timsousa3860 mesh generation is an option in the Instant NGP software, in fact. I'm not quite sure how they decide (what criterion), but I think it's basically a form of marching cubes. Traditional photogrammetry gives a mesh, though, so that's not particularly new. To me what's interesting is that NeRFs capture how light interacts with materials from different views. I'll be interested to see what evolves, as there are a number of approaches exploring ways of mixing traditional meshes, SDFs, volumetric representations, and more to capture a scene.
@@MrErichaines yes, you're 100% right
What I meant is that, at least right now, the results don't seem to be very accurate (specially on windows)
10:39 I want to be someone like you, you're awesome
Misleading title. This is not an animation, this is just camera movement. Animation. Something is animated. Nothing here is animated. This isn't a nitpick. Otherwise it's great. Like, seriously, thank you. But anyone looking for *an animated nerf* is going to be seriously disappointed.
Good point! Changed to "movie."
@@MrErichaines Thank you!