Hi Nick, Thanks for the great video. I don't know why but i cloned your repository and had to change one of my line code for it to work. Wanted to leave it as a reference for other developers const confidence = gesture.gestures.map( // had to change prediction.confidence (prediction) => prediction.score );
@@NicholasRenotte By using ev3dev, you can run Python or node on EV3. But I just run Fingerpose JS in my laptop then send realtime recognition result to EV3, so I can use my gesture as command to the Lego robot.
i have the same error "Cannot read properties of undefined (reading 'name') TypeError: Cannot read properties of undefined (reading 'name') at detect (localhost:3000/static/js/bundle.js:102:52)"
Fantastic work sir! I found your work while doing a bit of research to see if any of the makers/TH-camrs have been using hand gesture recognition to control their camera slider rigs. How cool would it be if we could control a 1 to 6 DOF CNC camera robot (linear, cartesian, gimbal, radial) by gesturing in the field of view with your hand(s)? I am thinking about all of my YT heroes that spend so much of their time making instructional videos to deliver the DIY/maker content we all love to gobble up. -Forefinger + middle finger +thumb triad for X,Y,Z,i,j,k targeting -'come here', and 'go away' gestures for zoom -focal distance control I love the honesty of your intro. I imagine myself as T. Stark designing the Mk2, but am painfully aware of my limitations. Thanks for letting me dream and flex my imagination!
Hey Nick, That was an amazing tutorial for gesture recognition right in the time of need. :) Just wanted to know how to shift the same codes to nodeJS and pre-existing bootstrap templates instead of React? I am currently working on similar stuff that recognizes hand gestures and controls the website but unable to proceed due to minimal knowledge of TensorFlow. Could you please provide your insights on the same?
Definitely, sounds like an awesome project @Shivani! You can embed the Tensorflow components on the client side and use them in a similar way to what I've done! These scripts allows you to bring TF and handpose into your existing bootstrap app :
Definitely @Bethany Wallace 🙌, you can train custom gestures (in this case for single handed poses) for basic sign language. I'm planning on creating a video on it in a few weeks time!
hi Nick ! that's is an awesome product , I was wondering if something like this exist with the body gesture ? I'd like to add extra feature on my stream when I'm on VR, so it could make action depending of my body movements ...
Very cool video! I have only one problem - if I want to add more emoji for detection from hand - where can I see the code for other hand emoji? Thanks!
@@offjoao5659 I did not include this line: const confidence = gesture.gestures.map( (prediction) => prediction.confidence ); const maxConfidence = confidence.indexOf( Math.max.apply(null, confidence) ); Then I replaced maxConfidence with 0 Before: setEmoji(gesture.gestures[maxConfidence].name); After: setEmoji(gesture.gestures[0].name); It worked fine to me, after that. Let me know if it worked for you.
@@melaniecasabar2183 Thank you, your answer has actually helped me solve something. I was looking at the documentation for the fingerpose npm module and found out that : prediction.confidence is actually: prediction.score so after a lot of debugging and lots of console logs later, this is the finished code: const confidence = gesture.gestures.map( (prediction) => prediction.score ); //console.log(confidence) //console.log(gesture) const maxConfidence = confidence.indexOf( Math.max.apply(null, confidence) ); console.log(maxConfidence) setEmoji(gesture.gestures[maxConfidence].name);
Hi Nicholas, Thankyou for the great tutorials thus far! I've managed to follow along with no errors during the first video, had most of this one but i've come across a error that results in it not logging to the console the correct gesture (but I can still see the detection of the correct pose if I check the console and the array. When I change my hand from to the gesture I lose my markers. I have checked my code against your git repo and I haven't found any differences (minus the extra stuff you added). This is the error: App.js:78 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'name') at detect (App.js:78:1) It's saying the name after the: setEmoji(gesture.gestures[maxConfidence].name); is undefined. When I check the console (as I mentioned it does detect) I can see the "name" in the array before the correct gesture. The only thing I can think of is the fact the video was made two years ago and something may have changed? any help would be super appreciated, as i'd love to finish the project. Thank you
hello Nick , I followed the whole process that you have explained in video but I'm getting an error like type error: cannot read properties of undefined (reading 'name') at detect . Can you help me out with this.
Hello I want a solution for the hand gesture for example if someone joining his hand for prayer the same thing to be seen on the wall and also need to know what device will help me to get this done.
Thanks Nicholas! It is another wonderful tutorial! May I ask a question that when I use the react useState, the canvas's drawing is quite slow and cannot match to the hand movement. Could you have any solution of that since I would like to add some simple canvas game and integrate to tensorflow hand checking, thank you!
Definitely @Jerry, i worked with some of the other subscribers on performance fixes. Check out the source code, it's now a lot smoother as we stop the model loading so many times: github.com/nicknochnack/GestureRecognition
@@NicholasRenotte Hi Nicholas, thank you for your information! the react useEffect is quite good for performance fixes and I have changed the code into the following for running requestAnimationFrame and It can run smoothly as well ~ However, I have a big problem that when adding a simple canvas animation (just a circle crossing the canvas). It is very slow and sometime is undefined... Could you help me to create a code for a circle crossing the canvas into this tutorial? Thank you very much ! //updated code for performance const requestRef = useRef(); const runHandpose = async () => { const net = await handpose.load(); console.log("Handpose model loaded."); async function frameLandmarks(){ requestRef.current = requestAnimationFrame(frameLandmarks) detect(net); } requestRef.current = requestAnimationFrame(frameLandmarks) }; useEffect(()=>{ runHandpose(); return()=> { cancelAnimationFrame(requestRef.current); } },[]);
@@jerrylee1657 definitely, what you'd need to do is define another function that runs as part of the setInterval. Then you can use the canvas to draw and redraw your circle across the page. Check this out: developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Basic_animations
Hi Nick! Great video. Is it possible for a live video of handwriting to show up similar to your emojis but as alphabets. Say one was to write the letter A, could the AI pick it up as so?
Hey Nicholas, Such a great video it is. Keep going!! I am trying to make App which simply help for tracking hand in the pool or any water surface who are drowning. Can you please help with a start and any further area I need to consider. thanks I am trying to use REACT.JS
@@NicholasRenotte I don't really know to solve it. One of my idea was yesterday to realise a tap gesture. Because the display is not 100% reliable (because it is only a camera, a stereo camera would be better) I look in the last 5 frames to see whether the index finger ☝️ was recognized there. If the current gesture is a fist ✊, then you typed. My new idea to detect the movement direction is to analyse where the position of one landmark is. I look to see if the old position value is larger / smaller. The change in position must be at least 5-10 pixels so that the small fluctuations of the hand are not so important. The principle is similar to my typing gesture. Worth a try right? 😊 And a big thank you to your videos, they are really good, and this project will also be immortalized in my homework for the university 🙈
@@joeckjoeck4321 that sounds like an awesome approach. I'm actually working on something right now that might be able to help you. It's in Python but should be able to hit the spot. Basically it's a two step machine learning model. This is how it works: Step 1: Webcam captures Poses, Facial Landmarks and Hand Gestures using Media Pipe Holistic, call this Model 1 Step 2: Images are saved and are manually classified for different body language and facial expressions Step 3: A secondary model, let's call this Model 2, is trained to match the landmarks for certain poses, facial landmarks and gestures to certain body language expressions Step 4: Bringing it all together, when detecting a new frame the landmarks from Model 1 are passed to Model 2 to detect overall body language (sad, happy, nervous, angry) You could take a similar approach, capture the last five (or more) frames and use it to classify a dynamic action. I started smashing out the code this morning and realised I could do so much more with it, as soon as it's done I'll shoot you the code!
@@NicholasRenotte That sounds good 😃 At the beginning I tried to use mediapipe but I had problems with the installation so that I look for another ways. My study is due to be handed in soon and can therefore no longer introduce any major changes. But for the future, however, I will definitely continue to pursue your implementation in privat. I also think it's great that you answer and that rather fast - a really nice service of you 😂🙈
Hey man ,great work but I want some help from you code: setEmoji(gesture.gestures[maxConfidence].name); I am getting an error from this line of code ,In that .name part . please help me with that. Uncaught runtime errors: × ERROR Cannot read properties of undefined (reading 'name') TypeError: Cannot read properties of undefined (reading 'name') at detect (localhost:3000/static/js/bundle.js:108:52) when I start the app this is showing this kind of error
Hey nick. How do we stop the handpose model from loading again and again. Because this is leading to the browser taking upto way too much memory space and crashing
Possibly @Aaditya Knv, I think it might be refreshing the whole application each time which is causing it to remount the model. I'm working on improving performance for the custom gesture video. Will shoot through an update as soon as it's there.
If I'm not mistaken, the runHandpose function is running every time the component re-renders when state changes. This is causing the setInterval to compound... If you put the runHandpose function inside a useEffect hook with no dependancies, it should only run once, when the component initially mounts.
@@ChromeCover91 YOU ARE A GOD AMONGST MEN! That was it, I think it was not only the setInterval compounding but also the fact that the handpose model was continuously reloading. I owe you a beer or coffee! @Aditya Knv, here's what you need to change (I've also updated the Github repository): 1. Import use effect // OLD CODE import React, { useRef, useState } from "react"; // NEW CODE import React, { useRef, useState, useEffect } from "react"; 2. Wrap runHandpose in a useEffect hook with no dependencies // OLD CODE runHandpose() // NEW CODE useEffect(()=>{runHandpose()},[]); 3. Speed up detection rate // OLD CODE- This is optional but makes the detections more seamless const runHandpose = async () => { const net = await handpose.load(); console.log("Handpose model loaded."); // Loop and detect hands setInterval(() => { detect(net); }, 100); }; // NEW CODE const runHandpose = async () => { const net = await handpose.load(); console.log("Handpose model loaded."); // Loop and detect hands setInterval(() => { detect(net); }, 10); // MAKE DETECTIONS FASTER };
Thank you very much for your response I want your assistant (I am doing a graduate project about arm robot ,I bought Kinect One and Adapter, but I can't write the code needed to Kinect to read hand signals only) please Help me
Heya @Shady, you just need to make sure the Kinect is available as a webcam, I don't have one so I'm going based on feel here but check this out: social.msdn.microsoft.com/Forums/en-US/af2ca1de-3de2-40dd-8694-4daadccbc487/pc-not-seeing-xbox-one-kinect-as-webcam?forum=kinectv2sdk
@@NicholasRenotte Thank you very much for your response , I have Kinect one Xbox ,Running it on my laptop is very normal, but it gives me a complete picture
Hi Sir, i'm ur subscriber from indonesia. Your content is very usefull for my study. But, now i'm in the last semester in college. I have to make a final product to finish my study. And i try to make a sign language with fingerpose. The sign language that i use is bisindo based on indonesian sign language. But when i tried to use this step based on ur video to my product i got some problem and i feel confused to solve that. But sorry before sir, would you help me to make another video how to make a sign language with fingerpose maybe 3 or 5 alphabet from bisindo. so , i can to learn how to make it from your video. Thanks sir I'm sorry before sir, if my english is so bad. Thanks you sir. God Bless You
Nick can you please hekp me build an app where a gestures will be used as an input to a quiz app I dont know how to start, I want to assign peace gesture to option 1, victory gesture to option 2 and based on this an app that takes quiz with gesture inputs please help Nick please!!!!!!!!!!!
Heyaa @Atif khan, once you've got the quiz app setup instead of using setEmoji like this: setEmoji(gesture.gestures[maxConfidence].name); You could make some of the changes below, change: const [emoji, setEmoji] = useState(null) to: const [quizResult, setQuizResult] = useState(null) replace: setEmoji(gesture.gestures[maxConfidence].name); with: if(gesture.gestures[maxConfidence].name == 'victory'){ setQuizResult(1) } else if (gesture.gestures[maxConfidence].name == 'thumbs_up' ){ setQuizResult(2) } // then you can pass the quizResult variable to your quiz app!
Hmmmm, given your use case, I'm wondering if real time gesture recognition is the right tool @Atif khan. You might be better off using an image classifier and taking a photo of the gesture the user wants to pass through. That way you can associate a single frame with a single answer.
@@NicholasRenotte I was trying to make some custom gestures, but facing problems. Can't understand how to add a new gesture description. It would be cool if you will share how you do that. Thanks
The main problem with your videos is that teach how to install libraries, (aka install hand pose, human pose), what if I want to create an animal pose estimation based on my poses, can you create a video on how to create your own datasets and install in a pose estimation model
Thanks Nicholas I wish you soon have 1M subscribers within a year
Hell yes 🙌 , I've got 185 more video ideas on my list (that's the real number) so there's no stopping soon!
@@NicholasRenotte that's great....
These videos are soooo helpful! Keep up the great work!
Thanks so much @Gracie Mitchell! Definitely, plenty more in the pipeline!
Hey Nicholas ! Your videos are awesome !
Thank you so much @SR Rahul!
Gran video! Gracias a ti tengo una idea de tesis para terminar mi carrera. Estoy muy agradecido
Hi Nick, Thanks for the great video. I don't know why but i cloned your repository and had to change one of my line code for it to work. Wanted to leave it as a reference for other developers
const confidence = gesture.gestures.map(
// had to change prediction.confidence
(prediction) => prediction.score
);
hi, thank you so much for this - i've been struggling with this bug for abt an hour and this rly helped! how did you figure this out?
thanks a lot dude
Yes! custom GESTURES video! 😂🙏
Got it on the list @Vikash Chand 👨💻!
please
Tried today with my Lego Ev3, works like magic!
Ooooooh sick, I always wanted an Ev3. You can code with them using Js?
@@NicholasRenotte By using ev3dev, you can run Python or node on EV3. But I just run Fingerpose JS in my laptop then send realtime recognition result to EV3, so I can use my gesture as command to the Lego robot.
@@blind_sauce that is amazing, you've just given me an excuse to buy one! YESSSS!
Amazing. You got an instant subscriber.
Awesome @Susanta, thanks so much 🙏, welcome to the team!
Keep up the great work!
Thanks so much! Definitely, plenty more to come 🙏
You're amazing! Thanks for this!
Good morning! I have a problem with 85th line code. google Chrome show error reading("name"). Can you help me
i have the same error "Cannot read properties of undefined (reading 'name')
TypeError: Cannot read properties of undefined (reading 'name')
at detect (localhost:3000/static/js/bundle.js:102:52)"
simply great video😁
Fantastic work sir! I found your work while doing a bit of research to see if any of the makers/TH-camrs have been using hand gesture recognition to control their camera slider rigs. How cool would it be if we could control a 1 to 6 DOF CNC camera robot (linear, cartesian, gimbal, radial) by gesturing in the field of view with your hand(s)? I am thinking about all of my YT heroes that spend so much of their time making instructional videos to deliver the DIY/maker content we all love to gobble up.
-Forefinger + middle finger +thumb triad for X,Y,Z,i,j,k targeting
-'come here', and 'go away' gestures for zoom
-focal distance control
I love the honesty of your intro. I imagine myself as T. Stark designing the Mk2, but am painfully aware of my limitations. Thanks for letting me dream and flex my imagination!
Thank you so much @Fausto, that sounds like an amazing use case! Oh I've got aspirations to build my own Jarvis one day 🤣
Hey Nick, That was an amazing tutorial for gesture recognition right in the time of need. :) Just wanted to know how to shift the same codes to nodeJS and pre-existing bootstrap templates instead of React? I am currently working on similar stuff that recognizes hand gestures and controls the website but unable to proceed due to minimal knowledge of TensorFlow. Could you please provide your insights on the same?
Definitely, sounds like an awesome project @Shivani! You can embed the Tensorflow components on the client side and use them in a similar way to what I've done! These scripts allows you to bring TF and handpose into your existing bootstrap app :
@@NicholasRenotte thank you nick. Would definitely try this :)
@@shivanirao8657 awesome, let me know how you go!
Great video! Do you think I could train similar technology to recognise some basic sign language?
Definitely @Bethany Wallace 🙌, you can train custom gestures (in this case for single handed poses) for basic sign language. I'm planning on creating a video on it in a few weeks time!
@@NicholasRenotte awesome thank you! I'm looking into this for a university project so I will be sure to come back and watch!
Awesome @Bethany Wallace!!
Heya @Bethany, still a little basic but hopefully can help you out: th-cam.com/video/pDXdlXlaCco/w-d-xo.html
hi Nick !
that's is an awesome product , I was wondering if something like this exist with the body gesture ? I'd like to add extra feature on my stream when I'm on VR, so it could make action depending of my body movements ...
Very cool video! I have only one problem - if I want to add more emoji for detection from hand - where can I see the code for other hand emoji? Thanks!
You can add other classes @Nikita, just add the images and parse the logic :)
@@NicholasRenotte Thanks!
Hi Man. I'm wondering if you're able to use any camera you want or if you need a specific camera.
Thanks Nicholas!
And what do you thing about applying filter to this work? Eg Kalman filter? I'm working on this but it's bugging me.
Hey Nick, do you think for gesture recognition better to use Kinect? could more efficient?
Interesting suggestion. If Kinect has motion tracking already it might be faster particularly if it's embedded!
this is awesome, how could i approach a drag and drop reactjs component using gesture recognition?
The .name method doesn't work, any ideas why
" Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'name') "
Did you fix it? I have tried various methods but I'm not sure what is the error here
@@offjoao5659
I did not include this line:
const confidence = gesture.gestures.map(
(prediction) => prediction.confidence
);
const maxConfidence = confidence.indexOf(
Math.max.apply(null, confidence)
);
Then I replaced maxConfidence with 0
Before:
setEmoji(gesture.gestures[maxConfidence].name);
After:
setEmoji(gesture.gestures[0].name);
It worked fine to me, after that. Let me know if it worked for you.
@@melaniecasabar2183 Thank you, your answer has actually helped me solve something. I was looking at the documentation for the fingerpose npm module and found out that :
prediction.confidence
is actually:
prediction.score
so after a lot of debugging and lots of console logs later, this is the finished code:
const confidence = gesture.gestures.map(
(prediction) => prediction.score
);
//console.log(confidence)
//console.log(gesture)
const maxConfidence = confidence.indexOf(
Math.max.apply(null, confidence)
);
console.log(maxConfidence)
setEmoji(gesture.gestures[maxConfidence].name);
@@melaniecasabar2183 Thank you for your help. I was having the same error and I applied your method and it worked.
@@melaniecasabar2183 thank you
For some reason the .name method doesn't work, any ideas why
" Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'name') "
Can someone please help out, I don't know why it doesn't work
Hey, same problem here! Haven’t found a solution yet :(
did you find a solution?
@@pauldoring3003 yeah, can’t remember what I did tho lol
@@cal1092 Could you maybe look it up for me? :)
Is it possible to implement this on a mobile phone and using phone camera to replace the webcam? What are the changes needed?
Sure can, would need a little rework to pick up the camera. I believe you could take a look at Tensorflow for React Native!
Can it detect simple waving, like just waving your whole hand back and forth?
Hi Nicholas, Thankyou for the great tutorials thus far! I've managed to follow along with no errors during the first video, had most of this one but i've come across a error that results in it not logging to the console the correct gesture (but I can still see the detection of the correct pose if I check the console and the array. When I change my hand from to the gesture I lose my markers. I have checked my code against your git repo and I haven't found any differences (minus the extra stuff you added). This is the error:
App.js:78 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'name')
at detect (App.js:78:1)
It's saying the name after the: setEmoji(gesture.gestures[maxConfidence].name); is undefined. When I check the console (as I mentioned it does detect) I can see the "name" in the array before the correct gesture.
The only thing I can think of is the fact the video was made two years ago and something may have changed?
any help would be super appreciated, as i'd love to finish the project.
Thank you
I should add I'm really new to coding, only having coding for 10 weeks so it could be something simple (most likely is) lol
have you figured it out? I'm having the same issue
Hey !! Please make a video on detecting custom gestures ? Like using it to detect sign language using the same concepts in the above video ?
Yep! Still coming, I'm doing a video on custom gestures this week using fingerpose!
@@NicholasRenotte Awesome 😁
@@NicholasRenotte Hey .. when will your video come ? Still waiting 😀
Heya @@LightningXBolttt, my schedule got knocked around a bit last week! Should be out next week 😁!
@@NicholasRenotte just got the notification of the video...Would try it as soon as possible..Thanks alot 😄
Yesss... Good Job. Thank You
No, I’m not talking about just swiping left or right to transition between screens.
How to do this :D...
Thanks so much @Meji Maru! Ooooh, that's a little more hardcore, let me have a think about that!
@@NicholasRenotte Thank you! It's my favorite TH-cam Channel. What a treat!
hello Nick , I followed the whole process that you have explained in video but I'm getting an error like type error: cannot read properties of undefined (reading 'name') at detect . Can you help me out with this.
I have the same issue, tell me if anyone has a solution
@nicholas ?!
have you figured it out? I'm having the same issue
@@melaniecasabar2183 no
Hello I want a solution for the hand gesture for example if someone joining his hand for prayer the same thing to be seen on the wall and also need to know what device will help me to get this done.
Possibly hook it up to a projector?
Thanks Nicholas! It is another wonderful tutorial!
May I ask a question that when I use the react useState, the canvas's drawing is quite slow and cannot match to the hand movement. Could you have any solution of that since I would like to add some simple canvas game and integrate to tensorflow hand checking, thank you!
Definitely @Jerry, i worked with some of the other subscribers on performance fixes. Check out the source code, it's now a lot smoother as we stop the model loading so many times: github.com/nicknochnack/GestureRecognition
@@NicholasRenotte Hi Nicholas, thank you for your information! the react useEffect is quite good for performance fixes and I have changed the code into the following for running requestAnimationFrame and It can run smoothly as well ~
However, I have a big problem that when adding a simple canvas animation (just a circle crossing the canvas). It is very slow and sometime is undefined... Could you help me to create a code for a circle crossing the canvas into this tutorial? Thank you very much !
//updated code for performance
const requestRef = useRef();
const runHandpose = async () => {
const net = await handpose.load();
console.log("Handpose model loaded.");
async function frameLandmarks(){
requestRef.current = requestAnimationFrame(frameLandmarks)
detect(net);
}
requestRef.current = requestAnimationFrame(frameLandmarks)
};
useEffect(()=>{
runHandpose();
return()=> {
cancelAnimationFrame(requestRef.current);
}
},[]);
@@jerrylee1657 definitely, what you'd need to do is define another function that runs as part of the setInterval. Then you can use the canvas to draw and redraw your circle across the page. Check this out: developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Basic_animations
Nicholas Renotte yes, thats what i need! Thank you for the information!
Anytime @@jerrylee1657 , let me know how you go. Pumped to see your game! Would love to see you share it here once you're done 😁!
Hi Nick! Great video. Is it possible for a live video of handwriting to show up similar to your emojis but as alphabets. Say one was to write the letter A, could the AI pick it up as so?
Hi Nicholas, Can we use this Hand pose and Finger pose in python:? If yes, then any leads to the resources?
Sure can, check this out: th-cam.com/video/vQZ4IvB07ec/w-d-xo.html the gesture control isn't implemented but can be done manually.
@@NicholasRenotte Thank You.
Here we goooooooooooooo
@Atif khan hell yesss 🔥🔥
Can You do about 3D Hand Pose estimation ?
Plssss !!!
Hey Nicholas, Such a great video it is. Keep going!!
I am trying to make App which simply help for tracking hand in the pool or any water surface who are drowning.
Can you please help with a start and any further area I need to consider.
thanks
I am trying to use REACT.JS
Thank you for sharing
Thanks so much @TingCheng!
awesome video nicholas, one thing, is there template of this code in python?
help~ confidence in line 71 can't be define, and the maxConfidence always -1
If we want to add more emojis, In which directory are we gonna add them?
Could you help with me the logic?
I tried but not working :(
Is there a way to detect gestures dynamicly, so that a move to left the app also says left?
Heya @Joeck, not with this technique, you could use action detection to do it though!
@@NicholasRenotte I don't really know to solve it. One of my idea was yesterday to realise a tap gesture.
Because the display is not 100% reliable (because it is only a camera, a stereo camera would be better) I look in the last 5 frames to see whether the index finger ☝️ was recognized there. If the current gesture is a fist ✊, then you typed.
My new idea to detect the movement direction is to analyse where the position of one landmark is. I look to see if the old position value is larger / smaller. The change in position must be at least 5-10 pixels so that the small fluctuations of the hand are not so important. The principle is similar to my typing gesture. Worth a try right? 😊
And a big thank you to your videos, they are really good, and this project will also be immortalized in my homework for the university 🙈
@@joeckjoeck4321 that sounds like an awesome approach. I'm actually working on something right now that might be able to help you. It's in Python but should be able to hit the spot. Basically it's a two step machine learning model. This is how it works:
Step 1: Webcam captures Poses, Facial Landmarks and Hand Gestures using Media Pipe Holistic, call this Model 1
Step 2: Images are saved and are manually classified for different body language and facial expressions
Step 3: A secondary model, let's call this Model 2, is trained to match the landmarks for certain poses, facial landmarks and gestures to certain body language expressions
Step 4: Bringing it all together, when detecting a new frame the landmarks from Model 1 are passed to Model 2 to detect overall body language (sad, happy, nervous, angry)
You could take a similar approach, capture the last five (or more) frames and use it to classify a dynamic action. I started smashing out the code this morning and realised I could do so much more with it, as soon as it's done I'll shoot you the code!
@@NicholasRenotte That sounds good 😃
At the beginning I tried to use mediapipe but I had problems with the installation so that I look for another ways.
My study is due to be handed in soon and can therefore no longer introduce any major changes. But for the future, however, I will definitely continue to pursue your implementation in privat.
I also think it's great that you answer and that rather fast - a really nice service of you 😂🙈
@@joeckjoeck4321 oh, shucks. Got some stuff coming on it soon so hopefully it helps! Anytime, if you need a hand just hmu!
Hey man ,great work but I want some help from you
code: setEmoji(gesture.gestures[maxConfidence].name);
I am getting an error from this line of code ,In that .name part . please help me with that.
Uncaught runtime errors:
×
ERROR
Cannot read properties of undefined (reading 'name')
TypeError: Cannot read properties of undefined (reading 'name')
at detect (localhost:3000/static/js/bundle.js:108:52)
when I start the app this is showing this kind of error
facing same issue.
Which algorithm you used here??
This is done using the HandPose model from Tensorflow.Js!
is it possible to run this on stored video instead of webcam? anyone pls guide
Can this be done on datasets using video frames?
Probably easier to do it with Python but it should work! You would need to have the vid uploaded and process the detections frame by frame!
@@NicholasRenotte we were planning to do it on video datasets for dynamic gesture recognition.
@@raneggg got it, would definitely check out doing it in Python using Media Pipe!
Hey nick.
How do we stop the handpose model from loading again and again. Because this is leading to the browser taking upto way too much memory space and crashing
If youre using chrome, You can notice the rise in memory taking up by browser in chrome task manager
Its happening only when we are displaying the gesture emoji. Is it because of useState?
Possibly @Aaditya Knv, I think it might be refreshing the whole application each time which is causing it to remount the model. I'm working on improving performance for the custom gesture video. Will shoot through an update as soon as it's there.
If I'm not mistaken, the runHandpose function is running every time the component re-renders when state changes. This is causing the setInterval to compound... If you put the runHandpose function inside a useEffect hook with no dependancies, it should only run once, when the component initially mounts.
@@ChromeCover91 YOU ARE A GOD AMONGST MEN! That was it, I think it was not only the setInterval compounding but also the fact that the handpose model was continuously reloading. I owe you a beer or coffee!
@Aditya Knv, here's what you need to change (I've also updated the Github repository):
1. Import use effect
// OLD CODE
import React, { useRef, useState } from "react";
// NEW CODE
import React, { useRef, useState, useEffect } from "react";
2. Wrap runHandpose in a useEffect hook with no dependencies
// OLD CODE
runHandpose()
// NEW CODE
useEffect(()=>{runHandpose()},[]);
3. Speed up detection rate
// OLD CODE- This is optional but makes the detections more seamless
const runHandpose = async () => {
const net = await handpose.load();
console.log("Handpose model loaded.");
// Loop and detect hands
setInterval(() => {
detect(net);
}, 100);
};
// NEW CODE
const runHandpose = async () => {
const net = await handpose.load();
console.log("Handpose model loaded.");
// Loop and detect hands
setInterval(() => {
detect(net);
}, 10); // MAKE DETECTIONS FASTER
};
How do I add five finger wave pose?
Heya @Peter, you could do it with custom gestures! Check this out th-cam.com/video/WajtPtLAg-o/w-d-xo.html
Thank you very much for your response
I want your assistant (I am doing a graduate project about arm robot ,I bought Kinect One and Adapter, but I can't write the code needed to Kinect to read hand signals only) please Help me
Heya @Shady, you just need to make sure the Kinect is available as a webcam, I don't have one so I'm going based on feel here but check this out: social.msdn.microsoft.com/Forums/en-US/af2ca1de-3de2-40dd-8694-4daadccbc487/pc-not-seeing-xbox-one-kinect-as-webcam?forum=kinectv2sdk
@@NicholasRenotte Thank you very much for your response , I have Kinect one Xbox ,Running it on my laptop is very normal, but it gives me a complete picture
@@shadim.tanani8368 complete or incomplete picture?
is this usable in vr games?
Possibly, I'm not super up to speed with VR development but I'm going to be giving integration into Blender and Unity a go soonish!
great job .. kindly please help in with the codes
Link in description 🙂
Hi Sir, i'm ur subscriber from indonesia. Your content is very usefull for my study. But, now i'm in the last semester in college. I have to make a final product to finish my study. And i try to make a sign language with fingerpose. The sign language that i use is bisindo based on indonesian sign language. But when i tried to use this step based on ur video to my product i got some problem and i feel confused to solve that. But sorry before sir, would you help me to make another video how to make a sign language with fingerpose maybe 3 or 5 alphabet from bisindo. so , i can to learn how to make it from your video. Thanks sir
I'm sorry before sir, if my english is so bad. Thanks you sir. God Bless You
Heya @Herdin, got a new vid on sign language coming soon! Stay tuned.
@@NicholasRenotte Ok Sir, thanks you so much. I'll be waiting. God Bless U Sir 👏
Nick can you please hekp me build an app where a gestures will be used as an input to a quiz app I dont know how to start, I want to assign peace gesture to option 1, victory gesture to option 2 and based on this an app that takes quiz with gesture inputs please help Nick please!!!!!!!!!!!
Heyaa @Atif khan, once you've got the quiz app setup instead of using setEmoji like this: setEmoji(gesture.gestures[maxConfidence].name);
You could make some of the changes below, change:
const [emoji, setEmoji] = useState(null)
to:
const [quizResult, setQuizResult] = useState(null)
replace:
setEmoji(gesture.gestures[maxConfidence].name);
with:
if(gesture.gestures[maxConfidence].name == 'victory'){
setQuizResult(1)
} else if (gesture.gestures[maxConfidence].name == 'thumbs_up' ){
setQuizResult(2)
}
// then you can pass the quizResult variable to your quiz app!
Oh thanks Nicholas I'll try today
Hmmmm, given your use case, I'm wondering if real time gesture recognition is the right tool @Atif khan. You might be better off using an image classifier and taking a photo of the gesture the user wants to pass through. That way you can associate a single frame with a single answer.
@@NicholasRenotte thanks Nicholas will try
@@NicholasRenotte Can you make a tutorial on this ?
Yeah man!
Yeahyaa @Igor Cojocaru, anything else you'd like to see?!
@@NicholasRenotte I was trying to make some custom gestures, but facing problems. Can't understand how to add a new gesture description. It would be cool if you will share how you do that. Thanks
Awesome, I've got coming soon!
You rule.
Awww shucks! Thanks @D2D.
Sir, tell me how you made yourself
Like this th-cam.com/video/xSElsMUqFqI/w-d-xo.html :)
Thx!
DIEGOOOO! Thanks so much as always @Diego Caumont 🙏!
Can you please send the source code ; please
Heya @Shady, here it is! github.com/nicknochnack/CustomGestureRecognition
cool
Thanks @yi hu ✌
The main problem with your videos is that teach how to install libraries, (aka install hand pose, human pose), what if I want to create an animal pose estimation based on my poses, can you create a video on how to create your own datasets and install in a pose estimation model
sexy