RTX Face Tracking in TouchDesigner - Tutotrial
ฝัง
- เผยแพร่เมื่อ 1 ส.ค. 2021
- Get access to 200+ hours of TouchDesigner video training, a private Facebook group where Elburz and Matthew Ragan answer all your questions, and twice-monthly group coaching/mastermind calls here: iihq.tv/Trial
If you’re a TouchDesigner Beginner, check out our TouchDesigner Tutorial Series! We teach you all the basics of how to use TouchDesigner, including mouse controls, hotkeys, parameters, the operator families, and more: interactiveimmersive.io/touch...
Face tracking used to be something you had to pay a lot of money for or use custom hardware for to get any kind of real-time & rich data. Now with machine learning it's easy and can be done on almost any hardware. In this video, Elburz takes you through setting up Nvidia's new Maxine AR SDK-based facial tracking inside of TouchDesigner with RTX graphics cards.
Great video and instructions! Showing how to instance with it was really helpful for me!
I'm glad to hear! In the next few videos, we'll be diving into more content creation so I'm sure there will be other tricks you'll enjoy :)
Awesome! Looking forward to it!
Thank you for the great video! I learned today that Windows 10 allows me to type "cmd" in the address bar of the folder I'm in, and it will run the CMD prompt from that folder. Super efficient, hope it helps someone.
Our pleasure, thanks for watching!
Also, great tip! Super useful way to optimize the workflow in Windows
15:12 made me chuckle XD XD. Thanks for the walkthrough!
Our pleasure! Thanks for watching :)
Hello - thank you for all the amazing videos which I've been following for a while now, and I am planning to subscribe to the full service - for some reason my render will not display the face-tracked image... one reason might be that I was originally trying to get the process to work within an existing scene that already had another render and camera (and it seemed to be trying to pull the information from there)... though now I can't get it to work in its own scene, or even in another file entirely. If there is anything you can suggest I'd be very grateful.
Hi Sarah, sorry about the delay responding! Can you give me a bit more information about what you're seeing when trying to run this in a new project? Do you see data moving in the CHOPs? Can you tell me about what kind of system you're on (mac/PC) and which GPU you have?
There's also non rtx facetracking with mediapipe quite handy
That's true, I've heard good things. I'll give it a try and see if we can make a short video about it as well.
I've always run mediapipe outside of TD and send the data in over osc. Fps kinda tanks when running it in TD. That being said I was running the Holistic model, which is a combo of all others so it might be a little much.
You do not need the mesh or to do the whole complicated part at the beginning! The Landmark points are separate and you can just switch it on in the Face Track CHOP.
Nice find! Seems like you can get the landmarks without needing the generate the mesh file at the beginning. There are other features that might be locked behind the mesh file, but that's great to know that you could easily get those landmarks without them :)
Yes, the mesh is for the face Track SOP so it now seams strange that the mesh is loaded into the CHOP and not the SOP!
@@TheInteractiveImmersiveHQ
Also it seams that we are both having an issue where the Face Track Landmarks are squashed horizontally making them not fitting our face when mapping them back over the Video Input.
Thanks for the tutorial, how can I achieve the same effect with Kinect?
Our pleasure! You can use a Select CHOP to isolate the face-related channels that the Kinect provides and then follow a similar technique to the video to generate an instancing effect. The number of channels is going to be different than what RTX Face Tracking provides, so the final outcome is going to look different than what you see here.
Weird ... I am not getting the face_model0.nvf file. I couldn't move further.
Can you copy and paste the command you were putting into the terminal here so I can take a look at it? Also do you see any errors in your command line when you're trying to generate the file?
Hi from Korea.
Thank you for your tutorial. I use Mac OS, and I haven't used terminal. How can I make face_model0.nvf? I googled how to use terminal and I tried to do it, but I'm not getting the face_model0.nvf file.
/facemesh in-ui-MacBook-Pro:~ 0in$ cd/Users/0in/Desktop/21-2\ 4학년/newmedia
-bash: cd/Users/0in/Desktop/21-2 4학년/newmedia/facemesh: No such file or directory
/facemesh ConvertSurreyFaceModel.exe -shape=sfm_shape_3448.bin -blend_shape=expression_blendshapes_3448.bin -topology=sfm_3448_edge_topology.json -contours=sfm_model_contours.json -ibug=ibug_to_sfm.txt -out=output-face_model0.nvf
-bash: cd/Users/0in/Desktop/21-2 4학년/newmedia/facemesh: No such file or directory
gim-yeong-in-ui-MacBook-Pro:~ 0in$
@@user-jm7ku3lp2f Hi there! Do you have a Mac that has an RTX GPU in it? I'm not sure if they've ever sold one, so we should make sure you have that before we continue.
Hello, I'm having some problems composing nvf, at that one step around 7:13 of your video. We really need your help!
We have successfully completed every step up to this point until here. When we paste the modified code into the "command prompt", an error is reported (here's a screenshot), so that we can't generate the nvf file. Many attempts have been made and still no success. We are eagerly awaiting your reply!
What error are you getting? I don't see a screenshot here. Do you also have an RTX capable GPU?
@@TheInteractiveImmersiveHQ big thanks for this!
Hello from Taiwan.
Many thanks for all effort you have put into the tutorial.
I have a problem of generating a right nvf file, I pasted the code "ConvertSurreyFaceModel.exe --shape=sfm_shape_3448.bin --blend_shape=expression_blendshapes_3448.bin --topology=sfm_3448_edge_topology.json --contours=sfm_model_contours.json --ibug=ibug_to_sfm.txt --out=output-face_model0.nvf" into command but the file it created called "output-face_model0" not "face_model0". And I inserted the file into facetrack chop of TD, and then it came out nothing.
I am not sure which steps I was wrong, could you point a direction for me?
Thank you so much!
I really appreciate your videos really helps me a lot in developing this software.
Hello, thank you for your tutorial. When I complete the "command prompt" step and enter TD, I encounter a problem. After I complete the facetrack setting, I do not detect face data, and "failed to load feature: the file could not be found" is displayed. I see that there are 8 files in the fasemesh of your tutorial, and I only have 7. I don't have the "face_model0" file. I can only put the "output face_model0" file into the "mesh file". I don't know which step I missed. Can you give me some suggestions? thank you
Can you copy and paste the command you put into command line here? My best suggestion would be to start the process from the beginning and pay close attention to what files you're downloading, where you're saving them, and running the commands exactly like I am.
@@TheInteractiveImmersiveHQ ConvertSurreyFaceModel.exe --shape=sfm_shape_3448.bin --blend_shape=expression_blendshapes_3448.bin --topology=sfm_3448_edge_topology.json --contours=sfm_model_contours.json --ibug=ibug_to_sfm.txt --out=output-face_model0.nvf
@@TheInteractiveImmersiveHQ ConvertSurreyFaceModel
expression_blendshapes_3448
ibug_to_sfm
output-face_model0
sfm_3448_edge_topology
sfm_model_contours
sfm_shape_3448
Thank you for your reply. I don't know which step I made a mistake...This is all the files in my folder.
@@TheInteractiveImmersiveHQ I solved it! I didn’t upgrade to the latest version of TD..thank you again!
@@7n454 No problem! That's usually a great first step especially with features that are so new. Enjoy!
Hi, thanks for your tutorial, but I have some problem composing nvf. it shows :
C:\Users\chenf\Downloads\TDFacemesh>ConvertSurreyFaceModel.exe --shape= sfm_shape_3448.bin --blend_shape= expression_blendshapes_3448.bin --topology= sfm_3448_edge_topology.json --contours= sfm_model_contours.json --ibug= ibug_to_sfm.txt --out=output- face_model0.nvf
No shape file was specified.
ConvertSurreyFaceModel [flags]
where flags are:
--verbose[={true|false}] produce verbose output
--debug[={true|false}] write extra debugging information
--shape=| specify the input SFM shape file
--blend_shapes= specify the input SFM blend shape file
--topology= specify the input SFM topology file
--contours= specify the input SFM contours file
--ibug= specify the input SFM ibug mapping file
--in= specify the input file *.nvf
--out= specify the output file *.nvf or *.json
--recompute_topology[={true|false}] recompute the topology
ERROR: The parameter has an invalid value
Can you give me some suggestions? thank you.
Hi, thanks for your tutorial. i've got a command issue : Unknow format for contours.
First, just want to check that you have a 20xx or 30xx series Nvidia RTX card - any cards not in the RTX 20xx series or 30xx series will not work (this includes 40xx series cards). Second, double check that you've downloaded all the correct files and followed the instructions from the wiki article: docs.derivative.ca/Face_Track_CHOP
Ok thank you for your answer, so it’s a série card issue on my side.
Hi I would like to use this face track to create camera follow to face effect. Is it possible to make it
Yes, definitely! Check out these forum posts for some suggestions and examples: forum.derivative.ca/t/camera-movement-live/364511
forum.derivative.ca/t/using-facetracking-to-zoom-into-faces/388757
Hi! Weird question, but when I click on any of the last 3 files in the Github, there is no button to download them... The first 2 did however, so I'm unsure if missing something or what. Actually the edge topology one has a download button but when clicked it just opens a bunch of code. Anyway if anyone knows what I'm doing wrong let me know!
On some files on Github it doesn't show the download link, what you can do is click on the "Raw" button to see the full contents, then you can copy and paste it into a text editor and save it to the appropriate name and file extension. Alternatively you should also be able to right click on the "Raw" button and select "Save link as..." and it'll function similar to directly downloading it.
@@TheInteractiveImmersiveHQ thanks for the reply! I’ll give that a shot later today. Great vid btw!!
@@AngusMacMinn Our pleasure! Keep me posted if that fixes the issue for you!
I'm getting an incompatibility error when trying to run the exe in command prompt -- anyone else experiencing this?
Hi Jordan, what hardware are you running on? Can you copy and paste the command you're running in command prompt as well as the error it's giving you?
Hi Jordan, I had the same problem, I went back an redownloaded the files and reran the command and it worked. Maybe a corrupted file or something.
Error: cannot create face detect feature: The GPU is not supported (/project1/facetrack1)........................................... Could you please tell me the solution?
Hi! From the error message, it looks like your particular graphics card doesn't support this functionality. The Face Track CHOP actually requires an Nvidia RTX graphics card to function.
doesn't Nvidia GTX graphic support to this? what is the solution ?
Unfortunately, the Face Track CHOP requires a 20 series or higher Nvidia GPU, so this particular effect is limited to RTX cards only.
If you're still looking for face tracking options that you can use with the GTX card, you might look for a used Kinect 2 and PC adapter. That might be a cheaper option than upgrading to a newer graphics card. Hope that helps!
hi got this error!!im running win10 home 64bit!! This version of C:\facetracking\ConvertSurreyFaceModel.exe is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.
Hmm that's interesting. I haven't seen that one before. Could you try downloading Nvidia Broadcast app and seeing if that works on your system or if it also gives you an error? That'll help us track down the issue
I am also getting this error, Windows is saying the application is "not compatible with 64x version of Windows.. Not sure why this is happening..
I installed Nvidia Broadcast and that works just fine.
is it possible for mac os?
Unfortunately it's not :(
The SDK the Face Track CHOP uses is only supported by Windows and an Nvidia RTX graphics card
sorry， I am not getting the face_model0.nvf file. I couldn't move further.
Have you followed the steps described at docs.derivative.ca/Face_Track_CHOP to generate the nvf file? If so, have you placed the conversion tool in the same folder as the Surrey Face Model files? Are you receiving any error messages?
Thank you. I have tried it again and it has been successful. I really like your video. Thank you again@@TheInteractiveImmersiveHQ
When I press enter after pasting the code., nothing happens, What am I doing wrong?
One thing to double check is that you've pointed the command prompt to the right directory before running the code (see 6:30 in the video). Also, if you're using the code from the video, make sure that you've got all of the required files in the same folder. If they're located elsewhere, you'll have to specify their location in the code you enter in the command prompt. Hope that helps!
And now is time for an RTX Laptop… my bank account will be not happy
Haha, very true!
Hallo.. i have problem when i open the ConvertSurreyFaceModel.exe, nothing happen, the aplication cannot open..do you have any advice. thanks
What error do you see in the command prompt? Can you copy and paste the command you're running here so I can double check it?
@@TheInteractiveImmersiveHQ sorry I mean, when I click the convert application (.exe), no window appears..
btw thanks for your response..please enlighten me. 🙏🙏
@@TheInteractiveImmersiveHQ it solve..thank you 🙏🙏
@@ockland23 Clicking on it won't do anything, you'll have to use the command line method I show in the video to use that exe. Sounds like you solved it :)
@@TheInteractiveImmersiveHQ yess...I used "cmd" to delete the "path" and it worked, but I forgot that my notebook VGA doesn't support....hehehehe..btw..thanks..
Hello! I really want to make a project for school with your setup, but I have a 2019 Macbook Pro with AMD Radeon Pro 5500M 8 Go
Intel UHD Graphics 630 1536 Mo. If I BootCamp windows on it, will I be able to work with the Face Tracking system? Thanks!
Unfortunately not because these processes specifically require Nvidia RTX GPUs, so they wont work on AMD hardware.
@@TheInteractiveImmersiveHQ Ok thank you!
Hello, I have a problem with run the command. I can not compose nvf. It shows:
"/sfm_shape_3448.bin": The file could not be opened
ConvertSurreyFaceModel [flags]
where flags are:
--verbose[={true|false}] produce verbose output
--debug[={true|false}] write extra debugging information
--shape=| specify the input SFM shape file
--blend_shapes= specify the input SFM blend shape file
--topology= specify the input SFM topology file
--contours= specify the input SFM contours file
--ibug= specify the input SFM ibug mapping file
--in= specify the input file *.nvf
--out= specify the output file *.nvf or *.json
--recompute_topology[={true|false}] recompute the topology
ERROR: The file could not be opened
I have follow every steps in your tutorial. (My Lap do have Nvida RTX.) (I was new in coding..) Could you please help me figured it out? Thank you!
Can you paste the command you were running as well? It seems like the file could not be opened so maybe something is wrong with the path in the command.
i have a similar problem :
C:\Users\SPARTAN PC\Documents\facemesh>ConvertSurreyFaceModel.exe --shape=sfm_shape_3448.bin --blend_shape=expression_blendshapes_3448.bin --topology=sfm_3448_edge_topology.json
"": Unknown format for contours
ConvertSurreyFaceModel [flags]
where flags are:
--verbose[={true|false}] produce verbose output
--debug[={true|false}] write extra debugging information
--shape=| specify the input SFM shape file
--blend_shapes= specify the input SFM blend shape file
--topology= specify the input SFM topology file
--contours= specify the input SFM contours file
--ibug= specify the input SFM ibug mapping file
--in= specify the input file *.nvf
--out= specify the output file *.nvf or *.json
--recompute_topology[={true|false}] recompute the topology
ERROR: The file has an unknown format
C:\Users\SPARTAN PC\Documents\facemesh> --contours=sfm_model_contours.json --ibug=ibug_to_sfm.tx --out=output-face_model0.nvf
"--contours" no se reconoce como un comando interno o externo,
programa o archivo por lotes ejecutable. ( it is not recognized as an internal or external command,
executable program or batch file
drive.google.com/file/d/17izmNESJStOQwp_1eRLVcb-9Xgp219Xy/view?usp=sharing in this link there is an screen shot,
How can i solve this , i have the hardware
@@apofis_media It looks like you're taking the single command and splitting it up between 2 different commands. It should start at the beginning where you start the command and end after --out=output but there shouldn't be any break between everything. Otherwise it will break like yours did.
@@TheInteractiveImmersiveHQ I get what you're sayin, how ever I don't get to understand how not to split it in two. i will look out if i can fix it. Thanks for the help and for answer me :)
@@TheInteractiveImmersiveHQSorry to bother, I just remind that i copied the code on text edit app, not on sublime text. Could be this the problem?