Face, Hand, Pose Tracking & More in TouchDesigner with @MediaPipe GPU Plugin
ฝัง
- เผยแพร่เมื่อ 17 พ.ค. 2024
- Hey! In this tutorial, we'll go over the new free open-sourced MediaPipe plugin for TouchDesigner that's GPU accelerated and works on Mac and PC with no instillation. The plugin supports face, hand, pose and object tracking with multi-person face detection, object detection, and image segmentation / background removal.
Link to the GitHub to download the Plugin: github.com/torinmb/mediapipe-...
Dom Scott's website: art.domscott.ca/home
Link to Torin's Patreon: / tblankensmith
Link to the body tracking alternative: • Tutorial 1/3 - New Bod...
Link to OBS for using video with MediaPipe on Mac: obsproject.com/
Link to SpoutCam for using video on PC: github.com/leadedge/SpoutCam/...
0:00 Intro and Overview
1:44 Project Setup
5:43 Face Landmark Detection
9:37 Hand Tracking + Gesture Recognition
13:49 Pose Tracking
16:37 Object Tracking
18:05 Multi Person Face Detection
19:21 Image Classification
20:56 Image Segmentation / Background Removal
23:58 On Mac Pipe Video into MediaPipe with OBS
26:26 On PC Pipe Video into MediaPipe with SpoutCam
28:30 Outro
Thanks for the plugin! For anyone having issues with the MediaPipe tox loading forever, the last version of 2022 (.35320) works for me on an M1 mac. I just avoided having to upgrade my commercial license for another $300 over this.
This is incredible! Thanks for publishing it ❤
Amazing, thanks for your hard work on this!
This is amazing, thank you both for making this!
This plugin is amazing, I hope you keep updating it. thank you so much for sharing it!!!
Wow this is incredible! I am really happy it also works for Mac computers!
you are the real master bro ! keep doing it !
WOW, I will be off grid now for some time to figure all of this out. Thank you. So helpful.
Thanks a lot for making this project public! it helps a lot for people that is starting
Sick sick sick. Am rolling into a project that might benefit from your great work. Perfect timing
you guys are amazing thank you so much!!!
Thanks for sharing such a helpful video! I never thought TouchDesigner could do this, and it's truly inspiring. Your video also has shown exactly what I've been looking for - how to superimpose TouchDesigner output to OBS.. Your effort is greatly appreciated!
o my, brilliant tracking without special hardware. great explanations.
this is phenomenal!!... i was about the tackle the same thing and wishing to find time to port everything to GPU.. now we have it.. amazing.
looks cool! thanks for sharing!
It's amazing! Thank you
Fantastic plugin-I think I can pack up my old Leap Motion Controllers now. Question: Is there a way to permanently have h1 be the left hand/h2 as the right? My experience now means that whichever hand is detected first becomes h1, regardless of whether it's my left or right hand. Specifically referring to the normalized data from Output 1 in the hand tracking component. Appreciate any advice on this, as well as the awesome tool.
Thanks for publishing it
We appreciate you!
This is unbelievable, thank you guys!
wow! I just get to know about MediaPipe - thank you so much for creating this and making this tools more accessible for individuals like me who have visual ideas but are not as technical as one can be.
Thanks! that manes a lot. Excited to see what you create with it!
More than amazing ! Thank you so much
Incredible content ! Verry smooth, everything is working perfectly ! (Macbookpro 2021 M1pro, lastest TD version)
you two are so friking awsome! thanks a lot !
This is incredible! Thank you so much
Hey, thanks! Question: so there is no way to keep up tp date with git pull atm? So I will have to manually download updates?
i would be really grateful to understand the spoutcam for windows in depth for the mediapipeline plugin. Thank you
How is it that you are able to have both the mediapipe node and video device in run at the same time? I can only ever have one working at a time because my webcam doesn't like it when multiple things try to use it at once. Is there any workaround?
Thank you. 감사합니다.
Awesome tutorial Torin!
I tried to do the obs set up but for some reason my virtual cam on the media pipe node gives me a black screen. I have tried various resolutions in obs preferences, the input video, my camera's resolution, and 1280x720 but it isn't working. Any recommendations?
Excellent tutorial and plugin. Very well made. I'm trying to apply the face feature points with a 3d model to animate the face according to the user. Any tips on how best to accomplish this? I keep running into blockers or dead ends.
how are you getting the z position of your fingertip from a non-3D camera? ty for sharing, this looks incredible - I've never had such a powerful tox up & running to quickly. Only issue was I had to download the latest TD.
thank you for your work it's real fun to use this and a lot of possibilities. I have one question: would it be possible to use the mediapipe plugin in td with an Infrared webcam?
Hey! Thank you for this. So great! Is there a way to upload videos instead of using the webcam?
Thanks so much for this amazing upgrade to working with mediapipe in TD, I was wondering if there is any benefit to using smaller resolutions than 1280 and if the spout method is the only way to achieve that downsizing? With the previous workflow of mediapipe I was able to get good hand coordinates at resolutions as low as 240x135 with an increase of roughly 8-10 fps. Although I am totally unfamiliar with the webbrowserCOMP and how/where it gets the webcam feed. Thanks
Hey! yeah the models run off a very small resolution feed, so there's no need to use a large video format. The only way to do it at the moment through spout or obs, which would add an additional delay
When looking for channel names(like around @20:37 & earlier) you can also just add a SELECT and click the black triangle next to "Channel Name" for a full list.
Thanks! i have a question!
can i add my hand gestures? (not something that already exists like Closed Fist, Open palm, Point Up ... )
Awesome. How is the performance of this vs the kinect?
WOW!
incredible
350£ kinect 2 is good for the bin!
still useful for pointcloud projects
Hi guys, first of all I wanted to thank you for this great tool! I am working on a project where, using my hands, I reconstruct images as if I were painting them and I wanted to ask you a question.
Is it possible to remove the neck part and keep only the hand part when working using output 1 of the image segmentation 21:33 (the body skin colour one)?
In the preview both hands, and part of the arms, are coloured pink/purple but as all the skin is detected the neck, which always remains uncovered, is also detected, ruining the effect. Thanks again and congratulations on everything!
nice job
Thank you so much! Such a game changer! Is there a way to out put the wire frame in a top? Trying to just out put stick man figure without the source video underneath it. Again, thank you!
Hey, this is a super great plugin. I have a question though, There's no way to keep a consistent left hand and right hand. Everytime I remove one hand from the frame, the remaining hand becomes Hand_1. This makes it impossible for me to create a controller using 2 hands :(. Does this have an easy fix or is this functionality coming in the future? :)
Looks amazing, but i could not get it to work, i get an error that resolutions is up to 720p. Tried putting the resolution down but still no working. Working on a MAc pro with sonoma TD 2022.33910
Hi! I got a problem, only loading and media pipe logo, can't detect the face in the camera, what is the solution?
cool Torin, thanks a lot 4 da work work work. , do you know if i connect 3 webcam to TD, can i use mediaPipe, using each one to track their onw user's hand?
Hey, thanks! Yes you can just duplicate the MediaPipe component, make sure to change the Websocket Port and the MediaPipe Port to something different than the original component and you can select a different web cam. You can do this with however many webcams you have. I'd recommend turning off all the models you're not using for speed
Hey Torin! Thanks for the great video and plugin!
I was wondering if there is a way to track multiple people at once. especially with the pose tracking? or is this not possible with MediaPipe?
Hey at the moment Media Pipe only has pose tracking for 1 person. It does look like multi-person pose is on the way soon though since their web based demo has multiple person detection(those examples are usually head of what they release to the public).
@@blankensmithing ok great thanks for your reply!
Diffrent question:
How can i use the Plugin with a higher resolution? I want to use the video output in 1080p but i can not use my Camera Input in a Video Device In TOP when i allready selected it in the MediaPipe Container.
@@user-uq9cn3wo8e The ML model scales down the webcam size, so using a higher resolution won't give better results. If you need to use a higher res cam you can use Spout Cam, or Syphon w/ OBS to change the resolution of your device and pipe it back into TD as a virtual webcam
what is the best way to use the face mask and replace it with a custom face model, IE make the face track onto a scary ghost or something fun instead of the base White mask. Trying to figure out the best way
Did you find a solution?
Hey guys! Anyone knows if there is a workaround to use Mediapipe TD for vertical video / stream?
Thank you so much for your plugin!!! I have a question about the bound box data, there is an "angle" from the mediapipe face bounding box value list, but in my project, it always shows "0".
Hey! which model are you looking at? Is it the face_detector, object_tracking, face_tracking component?
@@blankensmithing thank you for your immediate response! I'm looking for a face detector. I wanted to get the face/head rotation value, and then I saw an "angle" property, but it always shows"0".
Hi, I just would like to follow up on this if I did some wrong setting. Thank you so much🙏
Love the plugin! For some reason the latest version (0.32) shows the facePoints chop as an x,y,z chop (with green, yellow and red) as opposed to the time-slice style chop in the video. Any idea how to convert to the latter? Thanks!
u can try to use analyze CHOP i thought
@@luckuanfu2022 Perfect, thank you so much!
Can you connect cameras like arri?
Hi, I hope you can help me.
I just download the mediapipe plugin, so I open the main project MediaPipe TouchDesigner.toe then I Select my main camera and it doesn't show any image, but when i use a videodevice_in it automatically use my main camera.
Therefore almost every subsequent node show me some kind of error
Im using a macbook air M1, so, the main camera is a facetime camera.
Also, when I drag and drop some of the .tox at an empty Touchdesigner project it just doesnt drop at the workspace. Any idea if Im the one thats doing something wrong or someting else happening?
Hello.. does this work on TD non-commercial as well? I subscribed to your patreon and downloaded some TD sketches.. none of my webcams seem to work (just blank white page)and I am wondering whether it is the TD version! mediapipe downloaded from your Github has the same issue. My MacOS is 14.0 Sonoma. Thanks in advance.
Hey what version of TD are you using. You don't need a commercial license. It should run on the latest version of TD, but loading into much older version won't work
@@blankensmithing it's the most recent version. I downloaded it this week.
hello thank you so much for the plugin!!! I just have a problem of saving the file. Everytime I save my file and exit, when I open the file again, all the work I've put disappears.. if the file is not saved in the same folder of the toxes folder, the work is still there but it says the file is having a trouble loading components(probably looking for the toxes files) How can I solve this? :(
Hey if you've lost any progress just put the tox files in a folder called toxes that's in the same directory as your .toe file and re-open your .toe file. In the future there's a toggle you can turn off on all the mediapip tox files in the parameter page under Common called "Reload .tox on Start"
@@blankensmithing omg thank you so much!!! I tried turning the toggle off from mediapipe.tox file and still had the same problem, I had to actually turn off all the toggles from rest of the tox files in the tox folder and convert them to tox files instead of toe files to make it work!! Thank you so much💗💗💗
This is my first time working with a plugin on TD, I followed the steps like you said, and referred to the github page but it doesnt load up media pipe for some reason, I selected my Webcam, checked it as well but MediaPipe doesnt finish loading, what should I do?
I was facing a similar issue. On downloading the current version of TD (2022.33910), it worked well!
Hey please follow the steps here: th-cam.com/video/Cx4Ellaj6kk/w-d-xo.htmlsi=jzo-JCOVxF89jOpS&t=139 so I can help debug. Make sure to attach screen shots. If you see a red x on a component, hover over it and include a screenshot of the text that's displayed in the error message. Make sure to list what version of TouchDesigner you're running as well
@@blankensmithing Hi, so I was able to figure out the problem and now it works just fine. Excited to create with it. For anyone facing a similar issue, I used OBS studio instead of spoutcam for the webcam stream [PC] . In the video input of Mediapipe I used OBS Camera (which was scaled to 720p) and thats how it started to work. I created a new scene in OBS and simply added my webcam to the scene. Keeping OBS open I restarted TD and I could see "OBS Camera" as an input in the drop down. Hope this helps anyone facing this issue. Cant wait to get started with this plugin finally:) Thanks to the creators again!!!
@amanpathak2338 Hey! I'm glad you got it working :) Thanks for the tips so we can pass it along to other people
Hi! I have a problem, when i open a patch all connection from MediaPipe base in disconnected. And i have red circles with cross. How i can connect them again?
Hey please follow the steps here: th-cam.com/video/Cx4Ellaj6kk/w-d-xo.htmlsi=jzo-JCOVxF89jOpS&t=139 so I can help debug. Make sure to attach screen shots. If you see a red x on a component, hover over it and include a screenshot of the text that's displayed in the error message
@@blankensmithingThank you for you help! Now it works good on my Apple laptop, but on PC it stucks in loading Media Mapper when i connect usb webcamera.
I resized usb camera resolution in OBS to 720p and it works now! Thank you very much!✨💜
Why does mediapipe report an error when I use it?
Amazing! This is huge. How can I get Syphon to show up in the cameras dropdown? I'm using someone's proprietary OpenFrameworks Syphon App to get a KinectV2 working on my M1 Mac. I only get the video feeds tho so no CHOP data. Incorporating this mediapipe plugin into my workflow would nearly complete finally having a fully functioning kinectv2 in touchdesigner on apple silicon!
never mind! Using ndi.
Another option is NDI in NDI out..
I feel like your intro gets louder every vid and its a slow prank :D
haha It's always the same volume, but may have to do this now :)
Hi, thank you for this beautiful project, but I am having an issue, why the MediaPipe box is always on Loading mode, and it print an image of Loading. I tried reseting the process but still doesn't work for me. Can you help me? Thank you!