NVIDIA Instant Neural Graphics Primitives (nerf) Installation
ฝัง
- เผยแพร่เมื่อ 10 พ.ค. 2024
- This is a much more detailed walk through of installing NVIDIA NGP system which converts video into 3d data (obj). I am talking through the steps needed to install NVIDIA Instant Neural Graphics Primitives (NERF) in more detail than my previous videos and show the exact steps needed to run the code examples and create your own 3d NERF.
FFMPEG download
github.com/BtbN/FFmpeg-Builds...
Scripts
- Generate the images and training model-
python colmap2nerf.py --video_in "..\data
erf\bear\bear.mp4" --video_fps 2 --run_colmap --aabb_scale 16
--You can run this code to get more info
python colmap2nerf.py -h
-- Generate the 3d mesh from
"./build/testbed" --scene ./data/nerf/bear
NVIDIA - NERF NGP
github.com/NVlabs/instant-ngp
Visual Studio 2019
visualstudio.microsoft.com/vs...
CMAKE
cmake.org/download/
Cuda
developer.nvidia.com/cuda-too...
Python
www.python.org/
Optix
developer.nvidia.com/designwo...
Colmap
github.com/colmap/colmap/rele...
If you found this useful and want to show some appreciation you can buy me a coffee - www.buymeacoffee.com/dirkteucher
/ dirkteucher
0:00 Intro
0@57 Windows developer console (Required to compile cmake)
2:10 CMAKE version 3.22.3
2:50 Python
3:22 Git install or download zip
4:04 pip install requirements
6:42 ffmpeg
7:33 environment variables
10:30 cmake
12:03 error codes
12:47 Running included fox example
15:17 Fox success
16:37 Build with your own images
18:29 Explaining commands
22:52 Training with images
27:50 Build a bear
28:22 Success - ภาพยนตร์และแอนิเมชัน
![คณะขวัญใจ - นางสาวขวัญใจ [Official Music Video]](http://i.ytimg.com/vi/a8zn5pGlRUY/mqdefault.jpg)
Links and code commands are in the description
Great Video, really helped get setup quickly.
Thanks for the video Dirk! Have you seen 'nvdiffrec' yet? Apparently it's like photogrammetry, but it spits out a model and a complete PBR material set!
Yeah I did read through some of the code and example json files a few weeks ago and it looked to me like you feed it a 3d object. Which is like a snake eating its own tail. But I could be wrong about that.
So on the Nvidia Key note they make it seem like you only need 2-4 photos (from what I remember) and it will use AI to figure out the rest. Have you tried using only a few photos to create models? if so what were your results?
Yeah I have tried it with 5 - 20 images and it never works. The developers own instructions on github reccomends 50-150 images. The NVIDIA key note demo was a bit missleading as far as I can see.
The Nvidia Demo itself used only 3 photos. They need to be quality reference photos, though. The entire point is to create a scene with as few input data as possible to get a very good output. With photogrammetry you need 100 photos to produce a decent result.
I do not believe you are correct. Could you please point to where NVIDIA said they only used 3 photos ? In the documentation the developers suggest using 50 - 120 images. I have also tested this myself and not been able to get good results with a DLSR camera and 10 images. But 50-100 images consistently produces good results.
@@DirkTeucher really? I was under the impression that it used
i installed colmap also added in envirenment still when use colmap command is says colmap is not recognised
please help me
Hi dirk san
your explanation is great . I'm beginner of instant-ngf. However, since I am a 360 cameraman, I want to use 360video as the original material. In that case, can I use the same method for normal videos?
I have seen others use 360 footage in NERF/NGP so I think it is possible. If you have some footage and want to send me a link to it on twitter I will try it out and do a video. It might just be the camera settings need to be adjusted like the FOV.
@@DirkTeucher san
Thank you for your kind offer. I have a lot of 360 videos. From there we will send you aerial and ground video. Is 8K (or 4K) 30P h264 equirectangular projection about 30 seconds each good?
@@TPhotoworks Send both if you can. The 8k one might be too big to run on my rtx 3090 but i will try.
ok, I got testbed running, but the problem is that the viewport is playing the picture at 5 fps! And this is with dynamic resolution turned on, but if you turn it off, the frame rate drops to 0.5 frames per second!
I read that for correct operation, you need to install cuda toolkit 10.2 (for video cards up to the 20th line), but the problem is that in this case there is an error that I wrote about earlier. It is not clear where the problem occurs, on the side of the compiler, visual studio or api toolkit from nvidia
I dream of using this utility to upload models to blender, please help me!
@@lilbaph Without some sort of error code or video screen recording of the problem I will not be able to help you.
Also if you want to upload 3d models to blender then I recommend you use meshroom or reality capture th-cam.com/video/87h_P2BXEag/w-d-xo.html . NVIDIA Nerf exports 3d models with really completely unusable topology. NERF exports a volume of polygons with thousands of holes and polys in the center of the volume and you will get far better results with traditional photogrammetry.
NERF was also developed with the 3.xx series of GPU's so if you are using older hardware like the 1650 it is extremely slow and full or errors.
do you think we can export a 3d mesh from this software?
You can export a mesh in .obj format. But the quality is not close to what we can make with current photogrammetry software.
Hello, I can't under any circumstances successfully execute "cmake --build build --config RelWithDebInfo" 12:00. The compiler version is the same as in the video. The version of visual studio is the same, all the necessary add-ons and variables are set correctly (which I checked more than once). I made everything mirror in the same way according to several tutorials and nothing, nothing comes out!
What is your CPU and GPU ? And what error comes out? It does take quite a long time to compile. I have 24 cores and I think it took a good 10 minutes to compile.
@@DirkTeucher i7 2600, gtx 1660 6gb
12gb ram
>>Thanks so much for the tutorial! I get up to here:
C:\Windows\System32>CD C:\NGP\instant-ngp-master
C:\NGP\instant-ngp-master>cmake . -B build
-- Selecting Windows SDK version 10.0.22000.0 to target Windows 10.0.19044.
CMake Error at C:/Program Files/CMake/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:311 (message):
CMAKE_CUDA_ARCHITECTURES must be valid if set.
Call Stack (most recent call first):
CMakeLists.txt:11 (PROJECT)
>>I don't understand what it means when it says:
We also recommend installing CUDA and OptiX in /usr/local/ and adding the CUDA installation to your PATH. For example, if you have CUDA 11.4, add the following to your ~/.bashrc
export PATH="/usr/local/cuda-11.4/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-11.4/lib64:$LD_LIBRARY
>>I installed CUDA and OptiX in
C:\Users\LOCAL
And I dont understand where to enter those other commands. Thanks so much
Charles
You need to make sure that cuda has been added to your environment variables path. So 7:33 open up environment variables, edit path and make sure that "C:\Users\LOCAL\NVIDIA GPU Computing Toolkit\CUDA\v11.6" is added in the path in the same way as you can see here 7:52 and here 8:43 . Though I think this is added automatically when you install so you should not have to add Optix and Cuda to the system path.
Have anyone tried geo-referencing the result?
you can fix getting texture map , but i don’t know why the quality isn’t good
You can see vertex colors in the simulation but I have not seen a way to export a uv texture map along with the .obj. Have you managed to export textures?
@@DirkTeucher i didn’t try but in this video demonstrate that you can extract up map texture, watch until end
@@khaledbouzaiene3959 Did you mean to include a link to another video demonstration that shows how to export textures? I don't think NERF can currently export textures.
@@DirkTeucher th-cam.com/video/t06qu-gXrxA/w-d-xo.html iam sorry i didn’t past link i guess
you can bake to texture in blender
Thank you so much for this tutorial
Unfortunately, I have this error after cmake compiling
C:
gp\instant-ngp>cmake --build build --config RelWithDebInfo -j 16
MSBuild version 17.3.1+2badb37d1 for .NET Framework
MSBUILD : error MSB1009: Project file does not exist.
Switch: ALL_BUILD.vcxproj
How can I get past that?
Looks like you are running the command incorrectly to me .... possibly. If you can upload a video screencapture showing me what you are doing here on youtube I would be happy to help if I can.
@@DirkTeucher Thank you so much! I still don't know what happened, but after I restarted a few times, I tried recompiling from the start and then it was ok! Thank you so much though
I’m just wondering how come they don’t compile an executable for All of us to try .. this is a lot of backend stuff that needs to be installed.
Probably because it's not really very useful for many practical applications yet. Photogrammetry produces far better results along with usable data + assets.
@@DirkTeucher you really believe the output quality of this vs photogrammetry is not as good? Maybe I’m doing photogrammetry wrong then.. because this looks way better to me.
@@Instant_Nerf it seems to look better, yeah
@@Instant_Nerf Yes I am 100% sure that NERF produces worse quality 3d models than photogrammetry. I would also say that what NERF produces is different and unique.
If you export the 3d models from NERF with obj the topology is completely unusable in other 3d applications like Blender/Maya and the textures are only vertex colors which is extremely low resolution.
Now if you want to export video and not 3d files then the quality can be pretty good and it can produce 3d scenes that traditional photogrammetry would completely fail at doing. Like large vistas, vegetation, buildings with reflective windows, water. But that is still just a low quality video and you will get 10x better results just recording video with a drone for example.
The only advantage that NERF currently has is that once you have built the scene you can move the 3d camera around in in anywhere you like instead of being stuck with the original shot you did with a drone. But the quality is still not really usable for most applications and video still just produces better results. Some examples of what I am talking about - th-cam.com/video/y5K-afDrSwU/w-d-xo.html
Have you installed the vulkan sdk to have Dlss enabled? I’d like to speed up the process.
Looks like a 1080 doesn't have enough VRAM - after lots of installing and faffing about (insert sadface) lol
That is a shame. I did get results where i only used 720p images with 30 images that only used up 6gb of vram. It should be possible to get results with 8gb vram cards but yeah this tech drains vram like crazy 😞
@@DirkTeucher Hmmm... So, I was using a 4k video. I have gone in and changed the resolution of the outputted images but I still get the same result. It complains about the GPU architecture (61) and recommends I try 75 (20 series cards). I do have a 1080ti which of course has 3GB more VRAM but I have a feeling I'll still get the same result if it is just looking at architecture.
@@virtualrealityfun8869 thats odd, someone else mentioned that it worked on a 980 but was super slow but maybe i missunderstood their comment. I'm on a 3090 so its flys for me.
@@DirkTeucher Well I just reduced them to 720p (20 images) and got the same message. "FullyFusedMLP is not supported for the selected architecture61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+." Then immediately afterwards I get the failed with error CUDA_ERROR_OUT_OF_MEMORY. I need to look up the fullyfusedmlp part.
OK well, incredibly the fox example works just fine. I've checked and double checked the transforms file (compared the two) - nothing wrong with my own but it still doesn't work. I'll keep trying. Next test is just take a few pictures instead of using a video.
After -B build, I get this:
-- Building for: Visual Studio 16 2019
-- Selecting Windows SDK version to target Windows 10.0.19043.
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
-- The CUDA compiler identification is unknown
CMake Error at C:/Program Files/CMake/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:633 (message):
Failed to detect a default CUDA architecture.
Compiler output:
Call Stack (most recent call first):
CMakeLists.txt:11 (project)
-- Configuring incomplete, errors occurred!
I installed CUDA and made sure it is in the PATH. CMake -h, colmap-h, and ffmpeg -h all seem to work in the command window. Not sure what to do next.
Try compiling again with Cmake 3.22.4 . There is a link to their github on the cmake website where you can get that version of cmake. I am assuming cuda and the other applications and pip dependencies have been installed.
@@DirkTeucher I have gotten the same thing, and reinstalled cmake 3.22.4. the same error still comes up =/
CMake Error at CMakeLists.txt:11 (project):
Running
'nmake' '-?'
failed with:
The system cannot find the file specified
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CUDA_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
See also "C:/instant-ngp/build/CMakeFiles/CMakeOutput.log".
that's the error i get
I got it working, my optix was pointing to the wrong dir
when i run "cmake . -B build" it gave me this error
CMake Error at CMakeLists.txt:38 (message):
Some instant-ngp dependencies are missing. If you forgot the "--recursive"
flag when cloning this project, this can be fixed by calling "git submodule
update --init --recursive".
-- Configuring incomplete, errors occurred!
See also "C:/Users/Username/Desktop/Instant Nerf/instant-npg/build/CMakeFiles/CMakeOutput.log".
also I cant find the Microsoft VS code folder location to add it to the environment variables?
and im i supposed to make a bin folder inside the ffmpeg folder?
It sounds like you need to do the pip install 04:02 . So go into your root folder where the requirements.txt file is located with your developer console and then run that command pip install -r requirements.txt .
Do not worry about vscode that is optional so ignore that.
For ffmpeg I have added a link to the zip file in the description where you can download the standalone version. If you are on windows 10 you need the one named "ffmpeg-master-latest-win64-gpl.zip" . So download that and use that bin folder in your environment variable .
@@DirkTeucher alright thanks, ill test that out, see if it works.
But this just came up th-cam.com/video/5j8I7V6blqM/w-d-xo.html would be cool to get a tutorial of how to install this. seems like it NeRF and makes meshs.
@@mahmood392 Oooh thanks for the info I will check that out later tonight ! Looks pretty cool.
@@mahmood392 I looked at the code and from my very quick 20 minute scan it looks like they are using .obj files to do the training. So its not very useful for photogrametry.
hello Dirk Teucher,
I have the same issue to the very 1st comment.
when i run "cmake . -B build" it gave me this error
CMake Error at CMakeLists.txt:40 (message):
Some instant-ngp dependencies are missing. If you forgot the "--recursive"
flag when cloning this project, this can be fixed by calling "git submodule
update --init --recursive".
-- Configuring incomplete, errors occurred!
See also "C:/test/Nvidia_imageTo3D_Ai/instant-ngp-master/build/CMakeFiles/CMakeOutput.log".
the thing is, I did the pip command too like you mentioned already. Then run this cmake . -B build command. I still get the same error mentioned.
Could you help provide some advise how to solve this?
I did tried all the colmap -h and everything u've mentioned too. Still get this error.
Would be grateful if u could help on this thanks! :D
Looks like the error mesage is asking you to run "git submodule update --init --recursive" from the cmd. Do you have git installed? Have you run that command from inside the instant-ngp folder? EDIT - also what version of CMAKE are you using?
@@DirkTeucher Oh, silly me. I didn't know git is another software needed. Thanks for the help! Its working now. I'm not from a programming / technical background, most of these things confuses me sooo easily. haha
Side note, for some very odd reason, the downloaded the ingp zip file from git hub does not work on the dev cmd console for me. The error keep occurring.
The 2 things I did slightly differently from ur tutorial is, that I installed git (as mentioned from ur kind reply) and clone the repository from dev cmd console instead of downloading the literal file from git hub. :)
Thanks for the help, mate!
Very helpful, thumbs up to you!
@@wvfx9078 That's awesome ! ... and don't sweat it about not knowing about git. Its all easy when you know how and difficult when you do not. I was the same and asked questions all the time (still do). I also used to swear I would never learn programming as I thought it was boring af and I was far more interested in 3d design but I ended up really loving programming and mixing the two together :D love it
I have no idea why the zip download did not work I definitely installed it by zip once a few weeks ago but maybe I ran that command through git too so maybe git is required. Who knows with these things all it takes is a comma sometimes to stop the whole system from working hahah 😭
Happy nerfing to you W VFX
@@DirkTeucher Hello again, yesterday I was trying this on a weaker GPU on my main PC.
I tried setting this up again in a different machine with a slightly higher GUP series. But the machine has 2 different GPU and smthing is not working right. there are some errors that I cant solve. Would u be keen to have a 1 on 1 chat for this? Easier for my to screenshot or share my screen to show u whats happening.
Im soo lost on this :(
@@wvfx9078 sure thing ... can you dm me your discord id or video conference link on twitter? or drop it here? Also what is the exact gpu model you are trying to use?
Hello sir. Great video.
I did follow your instructions. Everything worked fine. Until I type : "./build/testbed" --scene data/nerf/fox
In the console I get the following messages:
INFO Loading NeRF dataset from
INFO data
erf\fox\transforms.json
SUCCESS Loaded 50 images after 0s
INFO cam_aabb=[min=[1.0229,-1.33309,-0.378748], max=[2.46175,1.00721,1.41295]]
INFO Loading network config from: configs
erf\base.json
INFO GridEncoding: Nmin=16 b=1.51572 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
INFO Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
INFO Color model: 3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
INFO total_encoding_params=13074912 total_network_params=9728
The fox, is always blurry and it never get transformed as in the video. What I'm doing wrong?
Sorry, I'm new to everything but I cannot see what I'm doing wrong.
I have a GTX 1070 and GTX 970. I set CUDA_VISIBLE_DEVICES to 0 and NVIDIA_VISIBLE_DEVICES to 0.
I tried everything with the versions specified on the Github page, then your video, still a bad result. The last time I building with Cuda 11.7, CMake 24.0
What I'm doing wrong? The fox is always blurry no matter how much I wait.
Can someone please help me? :(
Sorry I do not know what the "FullyFusedMLP " error fix is. A few people running the 10xx series graphics cards have had this error. Maybe this only works on 20xx series or 30xx series cards . There are a few "issues" that have been raised on the developers github with similar sounding error codes ... so you might want to create your own issue there for the developers to look at if they choose to do this.
@@DirkTeucher Thanks you so much for your prompt and detailed answer. I really appreciate.