Nicholas, Thanks for the well explained tutorial. You did mentioned that it is accurate for a relatively close distance. To make the plotting of the coordinate more accurate, you can crop the frame to an aspect ratio 1:1 before you send it to the interpreter. For example I have 640x480 webcam. I crop the image to 480x480. It seem to be more accurate, but you lose 80p on each side of the frame. Code I used: width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) print(width, height) cropTopX = int((width - height) / 2) cropBotX = int(width - cropTopX) roi = [(cropTopX, 0), (cropBotX, height)] cutout = frame[roi[0][1]:roi[1][1], roi[0][0]:roi[1][0], :] Use cutout to send to the interpreter for processing
@pirondinimarco You are correct, I used cv2.copyMakeBorder() to make the aspect ratio 1:1 with a black border for my project. I needed the bigger view for my project.
That's exactly what I meant with showing examples of the result before starting the tutorial. Helps a ton understanding what I'm going to learn before watching the whole video. Do you have any plans on creating a tutorial for recommender systems? For example TH-cam recommending videos to me that I could watch. Keep up the good work! :)
I love his tutorials. I'm having a problem when trying to load the model. I keep getting, an error message: "Didn't find op for builtin opcode 'RESIZE_BILINEAR' version '3' Registration failed.
@@NicholasRenotte I tried that, for some reason, it keeps restarting my kernel after I import TensorFlow. When I try to import tensorflow in a regular python file i get the error "illegal hardware instruction"
I see your video regularly. Although i am not following your code as right now i m not working on building models and python. But you are rocking and working so hard. Your video worths a lot. Still love to see your captions.
Hey Nicholas, Thanks for the in-depth video. I really learned a lot from this video. My question is, "how do I calculate the distances between two key points? eg. distance of right shoulder - to - left shoulder?. Thanks alot.
@@sasquelle I have seen few videos on how to measure face/hand real life distance from camera but I had never tried something like ur describing. For sure you need to measure this dist with measuring tape or sth (you need to make few maesures, for example 1 meter from camera, 2 meters, 3 etc) or you can do research on how to make deep learning model for that. Check Murtaza's Workshop - Robotics and AI's Hand Distance Measurement with Normal Webcam video, maybe you will figure it out how to modify his code to get the result you're looking for.
Every AI tutorial presented in your channel is well elaborated. I really appreciated that. Folks who watch your AI tutorials, should obviously give you a LIKE and SUBSCRIBE. By the way, I am wondering whether you will launch a full course of Data Science on your channel along with AI lessons on the horizon.
I really like the format , with clear goals and objectives and structured ( like s German way 😂) . Is movenet only implemented in python ? Is there a react native tensor flow version of it , or a tutorial series you plan to put out ?
Thx for the nice tutorial! I believe I found a small error in the drawing function. To transform from MoveNet normalized coordinates back to image coordinates you should take into account the padding of the image, which is not the case in the current example. As a result the y-scale does is not entirely correct.
I got it to work but the predictions don't get drawn accurately. I guess it has something to do with the aspect ratio (1280/720) and the padding when feeding it to the model. But I don't see you addressing it. The predictions for my eyes for example, get drawn accurate when my head is in the vertical middle of the frame. But they rise above my eyes when I lower my head, and the sink below my eyes when I raise my head. I can help myself with manipulating the ky value, but Im struggling to find the source of this problem which im really interested in thanks for the tutorial btw!
Since you have experience using both MoveNet and MediaPipe, could you advice me on what is the best solution in your opinion? Also, are either of these solutions scalable?
so connected with your tutorial and the way you write the code first and then explain it line by line, I have question did you work with OpenPose library? if you did what's the difference between these two?
Hi this tutorial is very informative!! I just have a question, how would you make this to detect certain poses. I just need it to detect between 4 different poses.
This video was very helpful to me...but i do have a question, how can i determine angles (like your previous video on pose estimation using mediapipe)?? eagerly waiting for your reply , thanks!
Hey, I know it maybe an anoing question, but there is a youtuber and he speaks into his mika and an AI is speaking them for him. I think its called speech to text and text to speach and I only found 1 viedeo on youtube, but its not live so, what I want is,I want to play games with my frinds and speak with them but like with an AI and I jusgt dont know, how its done…
Hello. I have one question, that is the difference between this video and your previous video "AI pose estimation with python and mediapipe" is to use deeplearning model movenet? could you tell me about the difference? plz
The task, pose estimation is the same between both models. This model however Movenet Lightning can make predictions a lot faster, meaning it's great for applications that need to track movement quickly.
Hey, great tutorial! It looks like even though you install tensorflow-gpu it's still running on the CPU since it's using TFLight. Things run fairly fast w/ lightning, but I'm wondering if you know of any way to actually get this model running on the GPU to speed things up?
Thanks again for your amazing tutorial, Just I was wondering how to calculate velocity or angular velocity from mediapipe, I followed your tutorial and calculated the angles but I can't figure out how to do that.
Hey there! How do we detect key points and connections in 3d (do we have to take z coordinate too? and is it possible with one camera) and will it improve our detection accuracy If we do so? I want to detect if the back is arched or straight...is it possible?
hi bro, thanks for your wonderful video. I copied your code, but I add the set_tensor command, interpreter.set_tensor(input_details[0]['index'], np.array(input_image)) # this line creates troubles print("Shape Image:", img.shape) # (1440, 2560, 3) at this stage img = tf.image.resize_with_pad(np.expand_dims(img, axis=0), 192, 192) appears the following error, can you help me to resolve, thanks. ValueError: Cannot set tensor: Got value of type FLOAT32 but expected type UINT8 for input 0, name: serving_default_input:0 please, help me, if you can, thanks
Yes, same error in my case as well. it is requested highly if anyone can help in this regard, please. A quick response will highly be appreciated. thanks
Hi. Thank you for your tutorial. But with your exact code from your GitHub, a powerful computer with GTX1650, and a HD webcam 30fps, I have a really bad estimation, it's not at all as reliable as yours. Tracking of points get lost, many points are flickering. Tried différent threshold value, nothing is fixing it. I can't understand why ! May it be the background that is not a green screen ? Thanks
I have error [ Cannot set tensor: Got value of type FLOAT32 but expected type UINT8 for input 0, name: serving_default_input:0 ] in the line [ interpreter.set_tensor(input_details[0]['index'], np.array(input_image)) ] can you help me fix it
thank you for the tutorial, i have a problem, may you help me? i want to movenet plot skeleton based on my joint's coordinate, and i want to modify output-tensor and location of key point. may you please help me?
Hello I need some help, during a certain time in the video with trying to get the keypoints from the camera, I get this error in my code, I need help please to anyone who may know the answer. ValueError: Cannot set tensor: Got value of type FLOAT32 but expected type UINT8 for input 0, name: serving_default_input:0
Dear Nicholas, thank you for your fantastic tutorials! If you look at 47:00 of this video, when the face is in the middle of the camera frame, the position of eyes and nose are correct, but when you go backward, from distance, at 47:25, the eye points and nose are not correct, they are around your neck and when I bring my head below center, the detected points come over my head, and this also happens for the legs because they are below the center of the frame, do you know why that happens?! because the code from TensorFlow does not have this issue, but it is very slow in my laptop (the code from TensorFlow), your code is very good but this issue happens, do you know why?!
Hello Nicholas, thank you so much for the great work! It saved me a lot of time to set up the programm. However, maybe you are already aware of this issue: The key points does not allign well to the image if the human is not perfectly in the middle of the image. I think, since most frame (resolution from the camera) does not exactly have the aspcet ratio 1:1, which the imput image for the MoveNet has, you can not easlily mutiply the relative keypoints (keypoints_with_scores = interpreter.get_tensor(...)) with the pixel coordinate from the image frame. Through the /tf.image.resize_with_pad/ the pixel arragment betwwen camera frame and input img (192x192) does not match exactly. In order to get the right assignment, you have to conduct an affine transform, where you have to compute the inverse of the affine transformation matrix from the fram image into the input img. Please refer this site stackoverflow.com/questions/73677854/movenet-pose-estimation-renders-inaccurate-keypoints.
@@NicholasRenotte Would really be interested to see a tutorial from you on it. Your explanations are amazing and it would be a tremendous help for me!
first of all you have good tutorials and you really explain it well but I need your help, at 23:37 when I tried to execute it this is my error ValueError: Cannot set tensor: Dimension mismatch. Got 192 but expected 256 for dimension 1 of input 0. I am not sure if its because of the tensorflow model that I download but I just followed your video
First, try restarting your notebook, this should free up your webcam if it's inaccessible. Second, if it's still not working double check you have the VideoCapture device number set correctly.
I wish all tutorials would use your format , it so obvious and user friendly to structure things
Nicholas, Thanks for the well explained tutorial. You did mentioned that it is accurate for a relatively close distance. To make the plotting of the coordinate more accurate, you can crop the frame to an aspect ratio 1:1 before you send it to the interpreter. For example I have 640x480 webcam. I crop the image to 480x480. It seem to be more accurate, but you lose 80p on each side of the frame.
Code I used:
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
print(width, height)
cropTopX = int((width - height) / 2)
cropBotX = int(width - cropTopX)
roi = [(cropTopX, 0), (cropBotX, height)]
cutout = frame[roi[0][1]:roi[1][1], roi[0][0]:roi[1][0], :]
Use cutout to send to the interpreter for processing
nice fix
@pirondinimarco You are correct, I used cv2.copyMakeBorder() to make the aspect ratio 1:1 with a black border for my project. I needed the bigger view for my project.
this is my 6th Renotte Tutorial. I hope to go thru all your amazing tutorials! :D
That's exactly what I meant with showing examples of the result before starting the tutorial. Helps a ton understanding what I'm going to learn before watching the whole video.
Do you have any plans on creating a tutorial for recommender systems? For example TH-cam recommending videos to me that I could watch. Keep up the good work! :)
You're the reason I did it!! Awesome suggestion 🙏 definitely, I don't have it planned soon but it's definitely on the cards!
I love his tutorials. I'm having a problem when trying to load the model. I keep getting, an error message: "Didn't find op for builtin opcode 'RESIZE_BILINEAR' version '3'
Registration failed.
@@infiniteimprovement try using tensorflow 2.4.1
@@NicholasRenotte I tried that, for some reason, it keeps restarting my kernel after I import TensorFlow. When I try to import tensorflow in a regular python file i get the error "illegal hardware instruction"
@@NicholasRenotte damn, giving you a like just for this
All of your tutorials are really awesome. I like the way you present and the flow.
I love the fact that you upload regularly so much.
Your tutorials are incredible. Thank you for making this information so easily available!
I see your video regularly. Although i am not following your code as right now i m not working on building models and python. But you are rocking and working so hard. Your video worths a lot. Still love to see your captions.
Thanks soooo much man 🙏🙏🙏
I got the code working! I had to put the #draw keypoints and #draw edges before #make detections code to make it run
It was amazing, I used this and changed my code with input frames to another target and I want to give the input coordinates with 32 joints
Amazing, just amazing. So what's the biggest difference that you have analyzed between mediapipe vs this one?
I was about to post the same question haha.
Same Question
+1
It's way faster and can run on just about any device! I've got a feeling they'll port this over to mediapipe soon!
Hey Nicholas,
Thanks for the in-depth video. I really learned a lot from this video. My question is, "how do I calculate the distances between two key points? eg. distance of right shoulder - to - left shoulder?.
Thanks alot.
it depends if u wanna get dist in pixels or real life distance
@@obi666 Real life distance. so what is the distance from the shoulder to the elbow as represented in real life
@@sasquelle I have seen few videos on how to measure face/hand real life distance from camera but I had never tried something like ur describing. For sure you need to measure this dist with measuring tape or sth (you need to make few maesures, for example 1 meter from camera, 2 meters, 3 etc) or you can do research on how to make deep learning model for that. Check
Murtaza's Workshop - Robotics and AI's Hand Distance Measurement with Normal Webcam video, maybe you will figure it out how to modify his code to get the result you're looking for.
Amazing as always 🔥
Thanks a bunch @Meet Vardoriya!!
Every AI tutorial presented in your channel is well elaborated. I really appreciated that. Folks who watch your AI tutorials, should obviously give you a LIKE and SUBSCRIBE. By the way, I am wondering whether you will launch a full course of Data Science on your channel along with AI lessons on the horizon.
your video is very good influence for my study thank you!!!!
I really like the format , with clear goals and objectives and structured ( like s German way 😂) . Is movenet only implemented in python ? Is there a react native tensor flow version of it , or a tutorial series you plan to put out ?
Great to hear about the js version . I tried to search for it but didn’t really find it !
Outstanding stuff!
Thanks a lot that is super useful for running on the edge
Great stuff . . .!, I can not keep up with all tutorials from you . . . :)
Absolutely terrific as usual 👍
Thanks a bunch @Varun!
Thx for the nice tutorial! I believe I found a small error in the drawing function. To transform from MoveNet normalized coordinates back to image coordinates you should take into account the padding of the image, which is not the case in the current example. As a result the y-scale does is not entirely correct.
Can you please explain how to do it
Exaclty, this only works in his case because his video is aspect ratio of 1. Otherwise the keypoints are inaccurate.
The BEST and right on the cutting edge!
I got it to work but the predictions don't get drawn accurately. I guess it has something to do with the aspect ratio (1280/720) and the padding when feeding it to the model. But I don't see you addressing it. The predictions for my eyes for example, get drawn accurate when my head is in the vertical middle of the frame. But they rise above my eyes when I lower my head, and the sink below my eyes when I raise my head.
I can help myself with manipulating the ky value, but Im struggling to find the source of this problem which im really interested in
thanks for the tutorial btw!
Hello mister, did you find how to fix it?
@@malikeaboss I dropped the project - tried a few things with skewing the points in different ways, but that wasn't too successful
Could you please zoom in ?! it is very small the font size, difficult to read!!
Wow pose estimation, can we extract BVH out of it?
Still gotta look into that for you! Working on it atm :)
Can you please suggest me how to create a custom dataset for pose estimation and which architecture is best to train pose estimation model.
Wooah, exactly wt i wanted thanks man
YESSS! So glad you're checking it out!
Since you have experience using both MoveNet and MediaPipe, could you advice me on what is the best solution in your opinion? Also, are either of these solutions scalable?
Both seem pretty good! Scalability would likely be dependent on underlying infrastructure and how it's deployed tbh!
Hi, first of all, this was a really wonderful workshop, thanks!! Can you also send some references regarding movenet model in tensorflowjs?
Could you do another tutorial of pose detection with LSTM and MoveNet as feature extractor?
Ooooh, we should get back to that! Yup!
@@NicholasRenotte that would be amazing! looking forward to this tutorial
so connected with your tutorial and the way you write the code first and then explain it line by line, I have question did you work with OpenPose library? if you did what's the difference between these two?
Haven't done any work with it yet but it apparently is quite fast and does multi person tracking whereas this can only handle a single person!
Great tutorial man big thanks!
*linux mint* Only adjust I had to make on was instead of a float32 I had to use uint8 to get it working runs just fine.
Update the Thunder Model works with float32 and is the faster lightweight version, lightning uses uint8 and is the heavier more accurate model.
Love your videos
Thanks sooo much! Means a ton!!
can u please do a video about action detection using Webcam
Can you train the model on your personal dataset?
For detection, sure can!
Good work!
Great!
Please remember your finance series ;)
Definitely, still on the list, wanted to do something a little diff this week :)
When you stand up, the points that should be on your head point to different places. How can we fix this?
Its better than mediapipe pose??
This is amazing
Yup, think we're sticking to YT!
Instead of live cam can we use input as video
Thanks again for your amazing tutorial!! Could you do one on adding face filters like the ones on snapchat for web? Thanks!
In the pipeline!
Hi this tutorial is very informative!! I just have a question, how would you make this to detect certain poses. I just need it to detect between 4 different poses.
yeah i'm also working on a similar project, did you do any progress, if yes could you share it?
@hellothere_howdy hi we ended up doing it with trackers instead.
@@leon3959 would you be able to send your code, i want to refer it..
Wow this one's smooth ❤️❤️
Ikr?!
This video was very helpful to me...but i do have a question, how can i determine angles (like your previous video on pose estimation using mediapipe)??
eagerly waiting for your reply , thanks!
Same process, just apply it to these keypoint structures!
What if we change the tensorflow lightning to tensorflow thunder , if i change it , it is giving dimensions error ?
Ooooh, I think it expects a different input image size. Might need to resize your input in order to feed it!
Hey, I know it maybe an anoing question, but there is a youtuber and he speaks into his mika and an AI is speaking them for him. I think its called speech to text and text to speach and I only found 1 viedeo on youtube, but its not live so, what I want is,I want to play games with my frinds and speak with them but like with an AI and I jusgt dont know, how its done…
Oh, like a voice cloner?
Hello. I have one question, that is the difference between this video and your previous video "AI pose estimation with python and mediapipe" is to use deeplearning model movenet? could you tell me about the difference? plz
The task, pose estimation is the same between both models. This model however Movenet Lightning can make predictions a lot faster, meaning it's great for applications that need to track movement quickly.
How do you train a custom model and predict with your own ground truth?
hi, can you help me to know to draw points on other shapes but not humans and thank you you bean very helpfull.
did you use runescape music at the beginning?
Hey, great tutorial! It looks like even though you install tensorflow-gpu it's still running on the CPU since it's using TFLight. Things run fairly fast w/ lightning, but I'm wondering if you know of any way to actually get this model running on the GPU to speed things up?
Hey! If we want to train MoveNet model for real time detections can we use the same process as mediapipe library and add lstm?
Sure can!
Can you extract the x,y,z coordinates of a certain keypoint using this particular algorithm ??? is it possible?
Thanks again for your amazing tutorial, Just I was wondering how to calculate velocity or angular velocity from mediapipe, I followed your tutorial and calculated the angles but I can't figure out how to do that.
Would need to calculate the change in arc length first and divide by the time frame you're looking for
@@NicholasRenotte Thank you very much
Hey there! How do we detect key points and connections in 3d (do we have to take z coordinate too? and is it possible with one camera) and will it improve our detection accuracy If we do so? I want to detect if the back is arched or straight...is it possible?
I believe you get a z coordinate (I'm not too sure of it's exact representation though). I haven't tried integrating it but it might improve accuracy!
hi bro,
thanks for your wonderful video.
I copied your code, but I add the set_tensor command,
interpreter.set_tensor(input_details[0]['index'], np.array(input_image)) # this line creates troubles
print("Shape Image:", img.shape) # (1440, 2560, 3) at this stage
img = tf.image.resize_with_pad(np.expand_dims(img, axis=0), 192, 192)
appears the following error, can you help me to resolve, thanks.
ValueError: Cannot set tensor: Got value of type FLOAT32 but expected type UINT8 for input 0, name: serving_default_input:0
please, help me, if you can,
thanks
Yes, same error in my case as well.
it is requested highly if anyone can help in this regard, please.
A quick response will highly be appreciated.
thanks
Same
Will it work for multi person detection
i get this issue
NameError: name 'draw_connections' is not defined
Hi. Thank you for your tutorial. But with your exact code from your GitHub, a powerful computer with GTX1650, and a HD webcam 30fps, I have a really bad estimation, it's not at all as reliable as yours. Tracking of points get lost, many points are flickering. Tried différent threshold value, nothing is fixing it. I can't understand why ! May it be the background that is not a green screen ?
Thanks
I have error [ Cannot set tensor: Got value of type FLOAT32 but expected type UINT8 for input 0, name: serving_default_input:0 ] in the line [ interpreter.set_tensor(input_details[0]['index'], np.array(input_image)) ] can you help me fix it
thank you for the tutorial, i have a problem, may you help me? i want to movenet plot skeleton based on my joint's coordinate, and i want to modify output-tensor and location of key point. may you please help me?
Can this be used for multiple person pose detection?
Nope, but this can: th-cam.com/video/KC7nJtBHBqg/w-d-xo.html
Hello I need some help, during a certain time in the video with trying to get the keypoints from the camera, I get this error in my code, I need help please to anyone who may know the answer.
ValueError: Cannot set tensor: Got value of type FLOAT32 but expected type UINT8 for input 0, name: serving_default_input:0
Dear Nicholas, thank you for your fantastic tutorials!
If you look at 47:00 of this video, when the face is in the middle of the camera frame, the position of eyes and nose are correct, but when you go backward, from distance, at 47:25, the eye points and nose are not correct, they are around your neck and when I bring my head below center, the detected points come over my head, and this also happens for the legs because they are below the center of the frame, do you know why that happens?! because the code from TensorFlow does not have this issue, but it is very slow in my laptop (the code from TensorFlow), your code is very good but this issue happens, do you know why?!
Same doubt
hello mister, did you find how to fix it?
@@malikeaboss unfortunately not, but I did not use face points , only shoulders and buttocks
Hello Nicholas, thank you so much for the great work! It saved me a lot of time to set up the programm.
However, maybe you are already aware of this issue: The key points does not allign well to the image if the human is not perfectly in the middle of the image. I think, since most frame (resolution from the camera) does not exactly have the aspcet ratio 1:1, which the imput image for the MoveNet has, you can not easlily mutiply the relative keypoints (keypoints_with_scores = interpreter.get_tensor(...)) with the pixel coordinate from the image frame. Through the /tf.image.resize_with_pad/ the pixel arragment betwwen camera frame and input img (192x192) does not match exactly. In order to get the right assignment, you have to conduct an affine transform, where you have to compute the inverse of the affine transformation matrix from the fram image into the input img. Please refer this site stackoverflow.com/questions/73677854/movenet-pose-estimation-renders-inaccurate-keypoints.
What about the speed of movements, is it possible?
Possible yes, I haven't done it yet though!
@@NicholasRenotte Would really be interested to see a tutorial from you on it. Your explanations are amazing and it would be a tremendous help for me!
first of all you have good tutorials and you really explain it well but I need your help, at 23:37 when I tried to execute it this is my error ValueError: Cannot set tensor: Dimension mismatch. Got 192 but expected 256 for dimension 1 of input 0. I am not sure if its because of the tensorflow model that I download but I just followed your video
Change the conversion to 256
how can we estimate the action ?
im using google colab and the part "2 - Make detections" is not working with me, my webcam do not open
First, try restarting your notebook, this should free up your webcam if it's inaccessible. Second, if it's still not working double check you have the VideoCapture device number set correctly.
do you happen to have JavaScript links for this ... just like mediapipe ????
Same link, just hit the JS tab here: tfhub.dev/google/tfjs-model/movenet/singlepose/lightning/4
How i may run this with GPU?
Check this out: www.tensorflow.org/lite/performance/gpu_advanced
Is it possible to apply the angle calculating from your media pipe video to the movenet frameworl?
Possible bro
hello Can I use tensorflow-gpu2.0 ? thank you!
Haven't tested it with 2.0.0.
@@NicholasRenotte Thank you for your reply , I had use it that is ok
What if I run it in Tensorflow 2.5.0??
It will any problem?
I think you should be fine!
Bro I love you~~~
Love you too bro!
can it detect multi-person pls answer sir
Nope
@@NicholasRenotte i'd thank you for this tutorial it is really easy to understand .i have a question how can i perform multi-person pose estimation
@@sarahjamaal2741 here you go: th-cam.com/video/KC7nJtBHBqg/w-d-xo.html
Do you have this model's paper?
Doesn't look like they wrote a paper for it, best is the model card: storage.googleapis.com/movenet/MoveNet.SinglePose%20Model%20Card.pdf
Thanks, we just found the website
31:20
Tutorial how to convert this keyspoints to 3d animation files as BVH please