RTX Background Removal in TouchDesigner - Tutorial
ฝัง
- เผยแพร่เมื่อ 11 ก.ย. 2024
- Get access to 200+ hours of TouchDesigner video training, a private Facebook group where Elburz and Matthew Ragan answer all your questions, and twice-monthly group coaching/mastermind calls here: iihq.tv/Trial
If you’re a TouchDesigner Beginner, check out our TouchDesigner Tutorial Series! We teach you all the basics of how to use TouchDesigner, including mouse controls, hotkeys, parameters, the operator families, and more: interactiveimm...
The Nvidia RTX GPUs have tons of amazing and easy to use features for your interactive and immersive art. One of the most used features is the ability to quickly and easily to a high quality background removal using almost any camera without the need of any fancy lighting, equipment, or green screen. In this video, we'll cover setting up RTX's Background Removal natively inside of TouchDesigner.
Really great - so clearly explained 👍🏽
Glad you liked it! Thanks for watching :)
you just saved our project! thank youuu
Glad it helped! Thanks for watching :)
Nice!!! The RTX background removal is pretty great. I've been testing out a few different ai tools to for segmentation. Such a fun area!
It's awesome how far this stuff has come in such a short amount of time! It's really exciting and so flexible that you can use it basically anywhere without much extra gear or environment setup.
Very good Tutorial love it
Thanks! Glad you enjoyed it :)
Here before 100k subs
Yesss! 🙌
Nice Erza!! Can u teach how to use touchengine to control ue4's actors paramet in realtime??. 🙏🙏🙏 Always grateful for share your knowledge w the comunity...
For sure! It's on our to-do list :)
I'd be curious to know if the 3 frame lag is different when using the Nvidia Broadcast virtual video-in device, which also offers an alpha channel
Great question! I'll give it a test and see. I assume at least 1 frame of lag will still be present in Nvidia Broadcast App, but where it falls in the 1-3 range would be interesting to see.
nice tutorial! it works on my machine, but the nvidia background dialog box doesn't show a GPU option to select. does it has something to do with using the non commercial version? I checked that TD uses the RTX GPU and not the Radeon that's also present :/
I can confirm that it does work in the non-commercial version, which RTX card do you have? The Nvidia Background TOP requires a 20, 30 or 40 series Nvidia RTX card to operate. It's also worth updating to the most recent version of TouchDesigner if you haven't already.
@@TheInteractiveImmersiveHQ thanks for answering! I have the last version of TD, and an RTX 3060 card. are you on the TD help forum? I've made a post over there with a screen capture, but no one answered yet =(
How do I put effects on just the body instead of putting a background behind?
Once the background has been removed, you can add a Threshold TOP afterwards, which will create a solid fill where the person is in the image. Then you can use a Composite TOP to add texture/effects. Hope that helps!
thx for sharing...i am a td newbie , so which version of touchdesigner has nvdia background top？i cant find it.....
You'll have to use one of the newest stable builds. I believe it was released only within the last few months. Which version are you on now?
would it work with live webcam or camera feed?
Yes! Elburz is using the RTX Background Removal effect on a live webcam feed in the video :)
This might be somewhat out there, but would this also work with NVIDIA Tesla or Ampere Cards?
Good question! I had a lot of trouble finding a definite answer to this. Closest I found was that the RTX features work on Ampere architecture, but I'm not sure if they'll work on the server side A100 cards that you're thinking about. I didn't find much for Tesla either. Do you have one that you're trying to use or looking at cloud solutions for hosting something like this?
@@TheInteractiveImmersiveHQ No, I guess we'll just have to try in the cloud. We currently have a project that performs few/one-shot-based training on videoframes. On the RTX we got onsite the training is just a notch too slow, so we were looking into cloud-solutions. The learning benefits a lot from good masks, where the background removal would come in. So I though, if we have to use a cloud VM, why not put TD into it, when it brings additional benefits, like good masks. - Also thx for the fast answer!
@@DrakeRubicon It could work just make sure to have some patience. A lot of the time the cloud systems with those virtualized GPUs don't come with the best drivers and packages in Windows, so you'll likely have to install a bunch of things to get it working. Paperspace is also a good option if you need Cloud infrastructure but want to continue to use RTX cards in an easy to use setup.
wow! very nice! is there a way to get the same result in an MacOS? Nvidia background is supported just in Windows
Yes, although you’ll need to work with external software for it to work. OBS (free) has a plugin that emulates the Zoom-style background removal (obsproject.com/forum/resources/background-removal-portrait-segmentation-virtual-green-screen.1260/), which you could apply to your webcam feed and then route the video into TouchDesigner. There are also paid options like XSplit VCam (www.xsplit.com/vcam) as well.
@@TheInteractiveImmersiveHQ thank yoooouuuu
AMAZING! I am trying it but doesn´t work well with 5 webcams at the same time in my installation. Can we change the pretrained model to another with a better performance in stand up human body segmentation? I was trying to change it by myself but i can´t.
Thanks! The default model it uses is best suited for a single person sitting in front of the camera, so that might be why you're running into issues. You should be able to use the Model Folder parameter to point to a different folder with compatible model files (see wiki page: wiki.derivative.ca/Nvidia_Background_TOP).
@@TheInteractiveImmersiveHQ wiki web page is not active. Can you tell us just a little bit more about best compatible models formats and some links to download them? I reallyapretiate sooo much you quick answer. There´s not so much easy info about this so... THANKS!
"Error: failed to load feature: There was a problem deserializing the inference runtime engine"
What can I do?
What RTX graphics card do you have? If you have an RTX4xxx series card, this operator is unfortunately not supported yet. This error will pop up when you have an unsupported card.
@@TheInteractiveImmersiveHQ 4080. OK thank you
This doesn't work with the 40 series Nvidia cards
Unfortunately yes, the Nvidia Background TOP doesn’t currently support 40 series cards :(
Anything new? this seems exact same as kinect again 3 years back!
It might look like that at the surface but it has MANY benefits. The keying is a lot better, tighter, and less noisy than the Kinect player index/depth map. It also works in many more environments that a Kinect would have trouble with like outdoors, environments with existing IR lighting, and it won't trip out if you're wearing clothes that happen to reflect IR (very common issue with fashionable jackets at events haha). Finally it doesn't need any special hardware, so you don't have to worry about buying kinects, dealing with the trouble and cost of extending their signal, etc. All of that becomes so much easier and more cost effective when you can use any regular RGB camera like a generic logitech webcam.
@@TheInteractiveImmersiveHQ Nice one thx for reply, I'm think who actually owns and RTX for now? i'm sure in long run will benefits you just explaned. I'll be sad to see the kinect go but cool simplification of process
@@TheInteractiveImmersiveHQ You don't think motion capture with sensors will always beat what the electronic eye can see and deduct. hardware over software sort off. Going off topic here
@@RusticRaver I think a lot of people are starting to get RTX cards. RTX 2060 came out at the beginning of 2019 so it's almost 2 years old now, meaning most people who bought a computer in the last 1.5 years are likely on an RTX card and basically anyone buying a system now is getting a capable RTX card. It's sad to see kinect go, but it's already kind of gone as far as I'm concerned. Kinect 2 is officially discontinued, and Kinect Azure I'm still not 100% sold on it being a full replacement, so at this point you're either using a new camera that's still only just a dev kit (azure), a camera that's been discontinued and you have to buy OEM adapters for (kinect 2), or you can use the RTX stuff, which is current and supported and being rolled out as moderately production ready. So I see it as the lesser of 3 evils haha
@@TheInteractiveImmersiveHQ I see , very interesting, I'm after a motion capture sort of body glove that would allow me interact in VR world like in real life. Best fun in Touchdesigner so far was kinect + vr interacting with particles, if we could have that control over particles minus the glsl headfuck, excuse my french, waw, so close to making small games in node fun way rather than c++ madness.
It won't work for me
Hi! Could you describe the issues that you're running into? One thing to note about this particular effect is that it uses the Face Track CHOP, which needs a computer running Windows and an Nvidia RTX graphics card for it to work.
If you don't have an RTX graphics card or are running Mac OS, you'll end up with an error saying something like "This node is not supported".
@@TheInteractiveImmersiveHQ Exactly what I got (Mac m1) Any work around for M1 to remove black / make transparent background? Thanks!