RTX Face Tracking in TouchDesigner - Tutotrial

แชร์
ฝัง
  • เผยแพร่เมื่อ 1 ส.ค. 2021
  • Get access to 200+ hours of TouchDesigner video training, a private Facebook group where Elburz and Matthew Ragan answer all your questions, and twice-monthly group coaching/mastermind calls here: iihq.tv/Trial
    If you’re a TouchDesigner Beginner, check out our TouchDesigner Tutorial Series! We teach you all the basics of how to use TouchDesigner, including mouse controls, hotkeys, parameters, the operator families, and more: interactiveimmersive.io/touch...
    Face tracking used to be something you had to pay a lot of money for or use custom hardware for to get any kind of real-time & rich data. Now with machine learning it's easy and can be done on almost any hardware. In this video, Elburz takes you through setting up Nvidia's new Maxine AR SDK-based facial tracking inside of TouchDesigner with RTX graphics cards.

ความคิดเห็น • 86

  • @atltvhead2840
    @atltvhead2840 2 ปีที่แล้ว +2

    Great video and instructions! Showing how to instance with it was really helpful for me!

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว

      I'm glad to hear! In the next few videos, we'll be diving into more content creation so I'm sure there will be other tricks you'll enjoy :)

    • @atltvhead2840
      @atltvhead2840 2 ปีที่แล้ว +1

      Awesome! Looking forward to it!

  • @slendah
    @slendah ปีที่แล้ว +1

    Thank you for the great video! I learned today that Windows 10 allows me to type "cmd" in the address bar of the folder I'm in, and it will run the CMD prompt from that folder. Super efficient, hope it helps someone.

  • @CalebHoernschemeyer
    @CalebHoernschemeyer ปีที่แล้ว

    15:12 made me chuckle XD XD. Thanks for the walkthrough!

  • @sarahbeauclerk4708
    @sarahbeauclerk4708 2 ปีที่แล้ว

    Hello - thank you for all the amazing videos which I've been following for a while now, and I am planning to subscribe to the full service - for some reason my render will not display the face-tracked image... one reason might be that I was originally trying to get the process to work within an existing scene that already had another render and camera (and it seemed to be trying to pull the information from there)... though now I can't get it to work in its own scene, or even in another file entirely. If there is anything you can suggest I'd be very grateful.

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว +1

      Hi Sarah, sorry about the delay responding! Can you give me a bit more information about what you're seeing when trying to run this in a new project? Do you see data moving in the CHOPs? Can you tell me about what kind of system you're on (mac/PC) and which GPU you have?

  • @sdhruv95
    @sdhruv95 2 ปีที่แล้ว +4

    There's also non rtx facetracking with mediapipe quite handy

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว +3

      That's true, I've heard good things. I'll give it a try and see if we can make a short video about it as well.

    • @atltvhead2840
      @atltvhead2840 2 ปีที่แล้ว

      I've always run mediapipe outside of TD and send the data in over osc. Fps kinda tanks when running it in TD. That being said I was running the Holistic model, which is a combo of all others so it might be a little much.

  • @ruthjarmanandjoegerhardt4853
    @ruthjarmanandjoegerhardt4853 ปีที่แล้ว +1

    You do not need the mesh or to do the whole complicated part at the beginning! The Landmark points are separate and you can just switch it on in the Face Track CHOP.

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  ปีที่แล้ว

      Nice find! Seems like you can get the landmarks without needing the generate the mesh file at the beginning. There are other features that might be locked behind the mesh file, but that's great to know that you could easily get those landmarks without them :)

    • @ruthjarmanandjoegerhardt4853
      @ruthjarmanandjoegerhardt4853 ปีที่แล้ว

      Yes, the mesh is for the face Track SOP so it now seams strange that the mesh is loaded into the CHOP and not the SOP!
      @@TheInteractiveImmersiveHQ

    • @ruthjarmanandjoegerhardt4853
      @ruthjarmanandjoegerhardt4853 ปีที่แล้ว

      Also it seams that we are both having an issue where the Face Track Landmarks are squashed horizontally making them not fitting our face when mapping them back over the Video Input.

  • @HAINILOVE
    @HAINILOVE ปีที่แล้ว

    Thanks for the tutorial, how can I achieve the same effect with Kinect?

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  ปีที่แล้ว

      Our pleasure! You can use a Select CHOP to isolate the face-related channels that the Kinect provides and then follow a similar technique to the video to generate an instancing effect. The number of channels is going to be different than what RTX Face Tracking provides, so the final outcome is going to look different than what you see here.

  • @user-jm7ku3lp2f
    @user-jm7ku3lp2f 2 ปีที่แล้ว +1

    Hi from Korea.
    Thank you for your tutorial. I use Mac OS, and I haven't used terminal. How can I make face_model0.nvf? I googled how to use terminal and I tried to do it, but I'm not getting the face_model0.nvf file.

    • @user-jm7ku3lp2f
      @user-jm7ku3lp2f 2 ปีที่แล้ว +1

      /facemesh in-ui-MacBook-Pro:~ 0in$ cd/Users/0in/Desktop/21-2\ 4학년/newmedia
      -bash: cd/Users/0in/Desktop/21-2 4학년/newmedia/facemesh: No such file or directory
      /facemesh ConvertSurreyFaceModel.exe -shape=sfm_shape_3448.bin -blend_shape=expression_blendshapes_3448.bin -topology=sfm_3448_edge_topology.json -contours=sfm_model_contours.json -ibug=ibug_to_sfm.txt -out=output-face_model0.nvf
      -bash: cd/Users/0in/Desktop/21-2 4학년/newmedia/facemesh: No such file or directory
      gim-yeong-in-ui-MacBook-Pro:~ 0in$

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว

      @@user-jm7ku3lp2f Hi there! Do you have a Mac that has an RTX GPU in it? I'm not sure if they've ever sold one, so we should make sure you have that before we continue.

  • @7n454
    @7n454 2 ปีที่แล้ว +1

    Hello, thank you for your tutorial. When I complete the "command prompt" step and enter TD, I encounter a problem. After I complete the facetrack setting, I do not detect face data, and "failed to load feature: the file could not be found" is displayed. I see that there are 8 files in the fasemesh of your tutorial, and I only have 7. I don't have the "face_model0" file. I can only put the "output face_model0" file into the "mesh file". I don't know which step I missed. Can you give me some suggestions? thank you

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว

      Can you copy and paste the command you put into command line here? My best suggestion would be to start the process from the beginning and pay close attention to what files you're downloading, where you're saving them, and running the commands exactly like I am.

    • @7n454
      @7n454 2 ปีที่แล้ว

      @@TheInteractiveImmersiveHQ ConvertSurreyFaceModel.exe --shape=sfm_shape_3448.bin --blend_shape=expression_blendshapes_3448.bin --topology=sfm_3448_edge_topology.json --contours=sfm_model_contours.json --ibug=ibug_to_sfm.txt --out=output-face_model0.nvf

    • @7n454
      @7n454 2 ปีที่แล้ว

      @@TheInteractiveImmersiveHQ ConvertSurreyFaceModel
      expression_blendshapes_3448
      ibug_to_sfm
      output-face_model0
      sfm_3448_edge_topology
      sfm_model_contours
      sfm_shape_3448
      Thank you for your reply. I don't know which step I made a mistake...This is all the files in my folder.

    • @7n454
      @7n454 2 ปีที่แล้ว +1

      @@TheInteractiveImmersiveHQ I solved it! I didn’t upgrade to the latest version of TD..thank you again!

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว

      @@7n454 No problem! That's usually a great first step especially with features that are so new. Enjoy!

  • @ruichen2699
    @ruichen2699 2 ปีที่แล้ว +1

    Hi, thanks for your tutorial, but I have some problem composing nvf. it shows :
    C:\Users\chenf\Downloads\TDFacemesh>ConvertSurreyFaceModel.exe --shape= sfm_shape_3448.bin --blend_shape= expression_blendshapes_3448.bin --topology= sfm_3448_edge_topology.json --contours= sfm_model_contours.json --ibug= ibug_to_sfm.txt --out=output- face_model0.nvf
    No shape file was specified.
    ConvertSurreyFaceModel [flags]
    where flags are:
    --verbose[={true|false}] produce verbose output
    --debug[={true|false}] write extra debugging information
    --shape=| specify the input SFM shape file
    --blend_shapes= specify the input SFM blend shape file
    --topology= specify the input SFM topology file
    --contours= specify the input SFM contours file
    --ibug= specify the input SFM ibug mapping file
    --in= specify the input file *.nvf
    --out= specify the output file *.nvf or *.json
    --recompute_topology[={true|false}] recompute the topology
    ERROR: The parameter has an invalid value
    Can you give me some suggestions? thank you.

  • @thaiphan3647
    @thaiphan3647 4 หลายเดือนก่อน

    Hi I would like to use this face track to create camera follow to face effect. Is it possible to make it

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  4 หลายเดือนก่อน

      Yes, definitely! Check out these forum posts for some suggestions and examples: forum.derivative.ca/t/camera-movement-live/364511
      forum.derivative.ca/t/using-facetracking-to-zoom-into-faces/388757

  • @inert1168
    @inert1168 2 ปีที่แล้ว +1

    Hello, I'm having some problems composing nvf, at that one step around 7:13 of your video. We really need your help!
    We have successfully completed every step up to this point until here. When we paste the modified code into the "command prompt", an error is reported (here's a screenshot), so that we can't generate the nvf file. Many attempts have been made and still no success. We are eagerly awaiting your reply!

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว +1

      What error are you getting? I don't see a screenshot here. Do you also have an RTX capable GPU?

    • @sofigarazurreta4646
      @sofigarazurreta4646 11 หลายเดือนก่อน

      @@TheInteractiveImmersiveHQ big thanks for this!

  • @RamirosLab
    @RamirosLab 2 ปีที่แล้ว +1

    Weird ... I am not getting the face_model0.nvf file. I couldn't move further.

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว

      Can you copy and paste the command you were putting into the terminal here so I can take a look at it? Also do you see any errors in your command line when you're trying to generate the file?

  • @wei-hawhuang9530
    @wei-hawhuang9530 2 ปีที่แล้ว

    Hello from Taiwan.
    Many thanks for all effort you have put into the tutorial.
    I have a problem of generating a right nvf file, I pasted the code "ConvertSurreyFaceModel.exe --shape=sfm_shape_3448.bin --blend_shape=expression_blendshapes_3448.bin --topology=sfm_3448_edge_topology.json --contours=sfm_model_contours.json --ibug=ibug_to_sfm.txt --out=output-face_model0.nvf" into command but the file it created called "output-face_model0" not "face_model0". And I inserted the file into facetrack chop of TD, and then it came out nothing.
    I am not sure which steps I was wrong, could you point a direction for me?
    Thank you so much!
    I really appreciate your videos really helps me a lot in developing this software.

  • @thomasantoinepenanguer2629
    @thomasantoinepenanguer2629 ปีที่แล้ว

    Hi, thanks for your tutorial. i've got a command issue : Unknow format for contours.

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  ปีที่แล้ว

      First, just want to check that you have a 20xx or 30xx series Nvidia RTX card - any cards not in the RTX 20xx series or 30xx series will not work (this includes 40xx series cards). Second, double check that you've downloaded all the correct files and followed the instructions from the wiki article: docs.derivative.ca/Face_Track_CHOP

    • @thomasantoinepenanguer2629
      @thomasantoinepenanguer2629 ปีที่แล้ว

      Ok thank you for your answer, so it’s a série card issue on my side.

  • @AngusMacMinn
    @AngusMacMinn 2 ปีที่แล้ว +1

    Hi! Weird question, but when I click on any of the last 3 files in the Github, there is no button to download them... The first 2 did however, so I'm unsure if missing something or what. Actually the edge topology one has a download button but when clicked it just opens a bunch of code. Anyway if anyone knows what I'm doing wrong let me know!

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว +1

      On some files on Github it doesn't show the download link, what you can do is click on the "Raw" button to see the full contents, then you can copy and paste it into a text editor and save it to the appropriate name and file extension. Alternatively you should also be able to right click on the "Raw" button and select "Save link as..." and it'll function similar to directly downloading it.

    • @AngusMacMinn
      @AngusMacMinn 2 ปีที่แล้ว +1

      @@TheInteractiveImmersiveHQ thanks for the reply! I’ll give that a shot later today. Great vid btw!!

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว

      @@AngusMacMinn Our pleasure! Keep me posted if that fixes the issue for you!

  • @JordanEspionage
    @JordanEspionage 2 ปีที่แล้ว +2

    I'm getting an incompatibility error when trying to run the exe in command prompt -- anyone else experiencing this?

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว

      Hi Jordan, what hardware are you running on? Can you copy and paste the command you're running in command prompt as well as the error it's giving you?

    • @mixarstudios
      @mixarstudios 2 ปีที่แล้ว

      Hi Jordan, I had the same problem, I went back an redownloaded the files and reran the command and it worked. Maybe a corrupted file or something.

  • @SusimanRinoshan
    @SusimanRinoshan ปีที่แล้ว

    Error: cannot create face detect feature: The GPU is not supported (/project1/facetrack1)........................................... Could you please tell me the solution?

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  ปีที่แล้ว

      Hi! From the error message, it looks like your particular graphics card doesn't support this functionality. The Face Track CHOP actually requires an Nvidia RTX graphics card to function.

  • @unveil7762
    @unveil7762 2 ปีที่แล้ว +1

    And now is time for an RTX Laptop… my bank account will be not happy

  • @SusimanRinoshan
    @SusimanRinoshan ปีที่แล้ว

    doesn't Nvidia GTX graphic support to this? what is the solution ?

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  ปีที่แล้ว

      Unfortunately, the Face Track CHOP requires a 20 series or higher Nvidia GPU, so this particular effect is limited to RTX cards only.
      If you're still looking for face tracking options that you can use with the GTX card, you might look for a used Kinect 2 and PC adapter. That might be a cheaper option than upgrading to a newer graphics card. Hope that helps!

  • @cyberdox
    @cyberdox 2 ปีที่แล้ว

    hi got this error!!im running win10 home 64bit!! This version of C:\facetracking\ConvertSurreyFaceModel.exe is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว

      Hmm that's interesting. I haven't seen that one before. Could you try downloading Nvidia Broadcast app and seeing if that works on your system or if it also gives you an error? That'll help us track down the issue

    • @Xanthaxgum
      @Xanthaxgum ปีที่แล้ว

      I am also getting this error, Windows is saying the application is "not compatible with 64x version of Windows.. Not sure why this is happening..

    • @Xanthaxgum
      @Xanthaxgum ปีที่แล้ว

      I installed Nvidia Broadcast and that works just fine.

  • @user-it5em1ez2u
    @user-it5em1ez2u 7 หลายเดือนก่อน

    When I press enter after pasting the code., nothing happens, What am I doing wrong?

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  7 หลายเดือนก่อน

      One thing to double check is that you've pointed the command prompt to the right directory before running the code (see 6:30 in the video). Also, if you're using the code from the video, make sure that you've got all of the required files in the same folder. If they're located elsewhere, you'll have to specify their location in the code you enter in the command prompt. Hope that helps!

  • @juniorvigneault7608
    @juniorvigneault7608 2 ปีที่แล้ว +1

    Hello! I really want to make a project for school with your setup, but I have a 2019 Macbook Pro with AMD Radeon Pro 5500M 8 Go
    Intel UHD Graphics 630 1536 Mo. If I BootCamp windows on it, will I be able to work with the Face Tracking system? Thanks!

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว

      Unfortunately not because these processes specifically require Nvidia RTX GPUs, so they wont work on AMD hardware.

    • @juniorvigneault7608
      @juniorvigneault7608 2 ปีที่แล้ว

      @@TheInteractiveImmersiveHQ Ok thank you!

  • @user-gl5jg2dv3t
    @user-gl5jg2dv3t 6 หลายเดือนก่อน

    sorry, I am not getting the face_model0.nvf file. I couldn't move further.

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  6 หลายเดือนก่อน

      Have you followed the steps described at docs.derivative.ca/Face_Track_CHOP to generate the nvf file? If so, have you placed the conversion tool in the same folder as the Surrey Face Model files? Are you receiving any error messages?

    • @user-gl5jg2dv3t
      @user-gl5jg2dv3t 6 หลายเดือนก่อน

      Thank you. I have tried it again and it has been successful. I really like your video. Thank you again@@TheInteractiveImmersiveHQ

  • @user-gu3bl9rs2z
    @user-gu3bl9rs2z ปีที่แล้ว

    is it possible for mac os?

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  ปีที่แล้ว

      Unfortunately it's not :(
      The SDK the Face Track CHOP uses is only supported by Windows and an Nvidia RTX graphics card

  • @ockland23
    @ockland23 2 ปีที่แล้ว +1

    Hallo.. i have problem when i open the ConvertSurreyFaceModel.exe, nothing happen, the aplication cannot open..do you have any advice. thanks

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว +1

      What error do you see in the command prompt? Can you copy and paste the command you're running here so I can double check it?

    • @ockland23
      @ockland23 2 ปีที่แล้ว

      @@TheInteractiveImmersiveHQ sorry I mean, when I click the convert application (.exe), no window appears..
      btw thanks for your response..please enlighten me. 🙏🙏

    • @ockland23
      @ockland23 2 ปีที่แล้ว +1

      @@TheInteractiveImmersiveHQ it solve..thank you 🙏🙏

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว

      @@ockland23 Clicking on it won't do anything, you'll have to use the command line method I show in the video to use that exe. Sounds like you solved it :)

    • @ockland23
      @ockland23 2 ปีที่แล้ว +1

      @@TheInteractiveImmersiveHQ yess...I used "cmd" to delete the "path" and it worked, but I forgot that my notebook VGA doesn't support....hehehehe..btw..thanks..

  • @luiluiwang9633
    @luiluiwang9633 2 ปีที่แล้ว +1

    Hello, I have a problem with run the command. I can not compose nvf. It shows:
    "/sfm_shape_3448.bin": The file could not be opened
    ConvertSurreyFaceModel [flags]
    where flags are:
    --verbose[={true|false}] produce verbose output
    --debug[={true|false}] write extra debugging information
    --shape=| specify the input SFM shape file
    --blend_shapes= specify the input SFM blend shape file
    --topology= specify the input SFM topology file
    --contours= specify the input SFM contours file
    --ibug= specify the input SFM ibug mapping file
    --in= specify the input file *.nvf
    --out= specify the output file *.nvf or *.json
    --recompute_topology[={true|false}] recompute the topology
    ERROR: The file could not be opened
    I have follow every steps in your tutorial. (My Lap do have Nvida RTX.) (I was new in coding..) Could you please help me figured it out? Thank you!

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว +1

      Can you paste the command you were running as well? It seems like the file could not be opened so maybe something is wrong with the path in the command.

    • @apofis_media
      @apofis_media 2 ปีที่แล้ว +1

      i have a similar problem :
      C:\Users\SPARTAN PC\Documents\facemesh>ConvertSurreyFaceModel.exe --shape=sfm_shape_3448.bin --blend_shape=expression_blendshapes_3448.bin --topology=sfm_3448_edge_topology.json
      "": Unknown format for contours
      ConvertSurreyFaceModel [flags]
      where flags are:
      --verbose[={true|false}] produce verbose output
      --debug[={true|false}] write extra debugging information
      --shape=| specify the input SFM shape file
      --blend_shapes= specify the input SFM blend shape file
      --topology= specify the input SFM topology file
      --contours= specify the input SFM contours file
      --ibug= specify the input SFM ibug mapping file
      --in= specify the input file *.nvf
      --out= specify the output file *.nvf or *.json
      --recompute_topology[={true|false}] recompute the topology
      ERROR: The file has an unknown format
      C:\Users\SPARTAN PC\Documents\facemesh> --contours=sfm_model_contours.json --ibug=ibug_to_sfm.tx --out=output-face_model0.nvf
      "--contours" no se reconoce como un comando interno o externo,
      programa o archivo por lotes ejecutable. ( it is not recognized as an internal or external command,
      executable program or batch file
      drive.google.com/file/d/17izmNESJStOQwp_1eRLVcb-9Xgp219Xy/view?usp=sharing in this link there is an screen shot,
      How can i solve this , i have the hardware

    • @TheInteractiveImmersiveHQ
      @TheInteractiveImmersiveHQ  2 ปีที่แล้ว +1

      @@apofis_media It looks like you're taking the single command and splitting it up between 2 different commands. It should start at the beginning where you start the command and end after --out=output but there shouldn't be any break between everything. Otherwise it will break like yours did.

    • @apofis_media
      @apofis_media 2 ปีที่แล้ว

      @@TheInteractiveImmersiveHQ I get what you're sayin, how ever I don't get to understand how not to split it in two. i will look out if i can fix it. Thanks for the help and for answer me :)

    • @apofis_media
      @apofis_media 2 ปีที่แล้ว +1

      @@TheInteractiveImmersiveHQSorry to bother, I just remind that i copied the code on text edit app, not on sublime text. Could be this the problem?