Depth Estimation with OpenCV Python for 3D Object Detection
ฝัง
- เผยแพร่เมื่อ 11 ก.ค. 2024
- Inside my school and program, I teach you my system to become an AI engineer or freelancer. Life-time access, personal help by me and I will show you exactly how I went from below average student to making $250/hr. Join the High Earner AI Career Program here 👉 www.nicolai-nielsen.com/aicareer (PRICES WILL INCREASE SOON)
You will also get access to all the technical courses inside the program, also the ones I plan to make in the future! Check out the technical courses below 👇
_____________________________________________________________
In this video 📝 we are going to see How To Estimate Depth with Stereo Vision in OpenCV Python with multiview geometry. we are going to combine pretty much all of the things that we have been through in the computer vision tutorial here on the channel. We will use mediapipe for face detection, calibrate our stereo cameras and then use multiview geometry to estimate the depth to the face we are detecting.
If you enjoyed this video, be sure to press the 👍 button so that I know what content you guys like to see.
_____________________________________________________________
🛠️ Freelance Work: www.nicolai-nielsen.com/nncode
_____________________________________________________________
💻💰🛠️ High Earner AI Career Program: www.nicolai-nielsen.com/aicareer
⚙️ Real-world AI Technical Courses: (www.nicos-school.com)
📗 OpenCV GPU in Python: www.nicos-school.com/p/opencv...
📕 YOLOv7 Object Detection: www.nicos-school.com/p/yolov7...
📒 Transformer & Segmentation: www.nicos-school.com/p/transf...
📙 YOLOv8 Object Tracking: www.nicos-school.com/p/yolov8...
📘 Research Paper Implementation: www.nicos-school.com/p/resear...
📔 CustomGPT: www.nicos-school.com/p/custom...
_____________________________________________________________
📞 Connect with Me:
🌳 linktr.ee/nicolainielsen
🌍 My Website: www.nicolai-nielsen.com/
🤖 GitHub: github.com/niconielsen32
👉 LinkedIn: / nicolaiai
🐦 X/Twitter: / nielsencv_ai
🌆 Instagram: / nicolaihoeirup
_____________________________________________________________
🎮 My Gear (Affiliate links):
💻 Laptop: amzn.to/49LJkTW
🖥️ Desktop PC:
NVIDIA RTX 4090 24GB: amzn.to/3Uc7yAM
Intel I9-14900K: amzn.to/3W4Z5Cb
Motherboard: amzn.to/4aR6wBC
32GB RAM: amzn.to/3Jt2XVR
🖥️ Monitor: amzn.to/4aLP8hh
🖱️ Mouse: amzn.to/3W501GH
⌨️ Keyboard: amzn.to/3xUGz5b
🎙️ Microphone: amzn.to/3w1F1WK
📷 Camera: amzn.to/4b4Ryr9
_____________________________________________________________
Timestamps:
0:00 - Introduction
3:05 - Multiview Geometry
7:29 - Stereo Camera Calibration
19:37 - Face Detection with Distance Estimation
Tags:
#StereoVision #OpenCV #Object3D #ComputerVision #FaceDetection #MachineLearning - วิทยาศาสตร์และเทคโนโลยี
![[MPD직캠] 베이비몬스터 직캠 8K 'FOREVER' (BABYMONSTER FanCam) | @MCOUNTDOWN_2024.7.11](http://i.ytimg.com/vi/ylRx8QOuGXo/mqdefault.jpg)
Join My AI Career Program
👉 www.nicolai-nielsen.com/aicareer
Enroll in My School and Technical Courses
👉 www.nicos-school.com
This channel should grow more. Amazing content on computer vision.
Thanks a lot man! Really appreciate it and u definitely help a lot
I had a question : In the code script you said that the value has to be multiplied by 205.8 to get the result in cm. What is the default unit of distance (the one seen in both frames) , and how do you find this multiplication factor?
It's very helpful for me, thank you!
Then, I have some questions about calculating depth
1. focal length
Is it possible to use focal length from camera matrix or new camera matrix?
2. can you explain the difference between camera matrix and new camera matrix
Hello there,
Could you please make a vid about your setup and how to prepare everything so I can start working through your vids.
Thanks in advance
It's a great lesson
You are really great and amazing
Thanks for this good tutorials 💙
Thank you very much! I really appreciate those nice words! 😃
Truly, indeed. I found that this is a kinda breakthrough for my project.
@@inheelee3779 thank you very much! Glad that u can use it
Is there a way to get the alpha value by software? I think after rectification, my alpha has changed since i am not getting proper distance values. (They are way off). Plus, what is the use of "f" as it is not used anywhere in the triangulation code.
hello thanks for your videos, I am trying object detect with yolov4 and with ros/gazebo simulation. But I have some problems in the calibration. after the calibration images turn to [ 0 0 0 .... 0 0] matrixes and load black screen. I can not understand how to debug this. And of course I made stereo calibration and I have stereoMap.xml without 0 0 0 0 0 (any things looking like yours)
Can you use this for custom object recognition? Like for example the face? And can it distinguish between a real human face with just a photo of the human face?
what is the camera model you are using to do stereoscopy?
Amazing work
Thank you very much! I really appreciate it
The videos you have posted are of excellent quality. It has helped me a lot!!! Thank you :)
Glad that i can help, thank you! :)
ThankYou :)
You're welcome!
Which stereo camera module I can use with Raspberry Pi Zero 2 W ?
D435i show us the depth, pixel x, y. Why explain about two camera? It must to be considering post-processing getting datas from D435i? Or user just using the depth, pixel x, y?
Have to considering the relative values and applied to code?
Hey great vid. How does it fair with more than one person in the frame? I done a similar project for my final year project and the person detection model would detect the people in the left and right frames in different orders and it would throw off the distance measurement.
It should work fine but u would need to keep track of the people with an id assigned to them when coming into the frame and then u will just have an id with a corresponding depth
Hi sir, appreciate so much your videos about stereo vision, I was wondering if you can respond me just two questions:
1. What is your CPU power in this experience?, I mean processor, RAM, memory, etc.
2. What do you think is the minimum CPU power required to run this sketch and have a good performance?, my PC is more for ofimatic.
Look forward to your reply sir, thank you for the videos.
Which camera are you using? And what about the waveshare IMX219-83 Stereo Camera, the camera don`t create a pointcloud?
When you are doing the calibration is the baseline 9 cm which what you set in 20:36? How would I go about finding the focal length?
you can usually get the focal length from the camera manufactor. the baseline you measure in real life,is the distance between the two cameras
Hey thanks for the tutorial!! I am experiencing some issues with it though i hope you can help.
When i run the stereo_calib code, my images from both the cameras are rendered useless.. I tried calibrating my cameras one by one and got good results( 0.08 error). But when i try to do stereo calib using the same setup, it just doesnt work..
Hi thank you very much! Did u use the latest code from GitHub? When u load in the images with blob for stereo calibration u will need to sort the images or i will give very odd results like u mention
Thank you for the great videos. At what point does the depth accuracy drop? Is the distance between the cameras a driver of the depth distance?
Yeah the baseline distance plays an important role in the accuracy of the depth but also how far/close u can get accurate values for
@@NicolaiAI got it. So the resolution/quality of the camera/sensor can also play a big role?
Once again, thank you for great videos, been binge watching! Will start implementing some soon!
@@twiincentral8780 the resolution of the camera is the most important factor since u want as accurate disparities as possible. Thanks for watching! Glad that it can help u
Do you have a youtube clip where you discuss how to get a two-camera setup? Did you buy that or build it yourself? Do you have any recommendations?
Thanks.
Hi great idea, i should do a video about stereo cameras or how to setup on urself. In these videos i just use two webcams. Thanks a lot
@@NicolaiAI Cheers for the quick reply. Do you have good/accurate results with that setup or is that why you need good calibration? :) I am looking to do a project that includes depth-perception. I saw your clip about using transformations on a 2D image where you used a picture of a snooker table. I will definitely have a try at that and use the transformed 'bird-eye view' to see if I can estimate the position of objects such as the balls on the table. But I am also interested in stereo vision to see how that works and how it compares, accuracy-wise. I'm also thinking about picking up a cheap Kinect detector to see what I can do with that. Interesting subject!
It is really interesting and u can create some really nice things with it. If u do a project where u need accuracy I'll definitely recommend to calibrate the cameras. Also because it's just run a script and that's it. A Kinect would also be a good idea and good accuracy but from my pov i think its cooler with stereo camera if it's just for hobby 😃
Are your webcams just bolted onto a frame next to each other? Just wondering what is the best way. Of course part of the fun will be me trying it out.
I just kinda place the cameras and measure the distance between them. I could improve the accuracy from those videos if it was necessary
It's a great video but I have some issues though i hope you can help.
How can I estimate depth if there are multiple objection of an classification in the frame ? ( ex: I have 3 cups in the frame how can i know depth of each cup).
Sorry for the late reply. U can find the depth to each individual object if u are able to detect it. Then u can just directly copy this code and use it. In this video i onyl do it for 1 object/face
I am working on the similar project where I need to find depth of the facial landmark points.
Can you guide me how to find distance(depth information) with single camera? with single depth camera (using RGB image and depth image)
U can just take the index in the depth image of the facial landmarks
Hi! I love your videos. Just a question: how do I find the focal length and alpha at 20:50?
Thanks a lot! Those are physical camera parameters, can either be found in camera specifications or calibrated. Focal length and FOV
@@NicolaiAI When you say it can be found by calibration, does that mean I can calculate for the focal length in mm from the calibration matrix obtained? Which of the calibration values obtained do I use and what equation to get the focal length? Likewise, can I calculate for the exact FOV based on the calibration data obtained?
Great tutorial pulling all aspects of CV together. You mentioned GitHub link for the code base, is that only if you register for the course?
Thanks a lot! Most of my code is available on GitHub. Only the course code is not in there
@NicolaiAI Thanks for the response, where is the link for the code and am I permitted to use it for my project with changes to fit my application
@@NicolaiAI Also, are you available for consulting, if so please let me know I can set up time to discuss my project and help I need
Yeah i do. U Can reach out to me on mail thecodinglibraryyt@gmail.com
Then I’ll find the code on GitHub aswell
Can I replace the face detection for a sign detector? I want to use this for measuring depth of signs. The car will be moving, so I'm not sure if this is going to work
It depends on the depth sensor in the camera. Contact the manufacturer of the selected camera and tell us your insights. If the camera is behind the windshield it could be tricky to detect the reflected signal
Very useful video but I do have a question, and I'm sorry if you explained this and missed it, but would this work if there are multiple faces in the images?
Thanks a lot! If you have multiple faces in the image u might have to apply tracking on top of it so have the same mappings. You could maybe also just sort all the bounding boxes in the x axis and then match those from left to right directly
Negative values:
I have trained an object recognition model (using tensorflow api) and would like to determine the depth of the recognized objects (using the centers of the bounding boxes of each object as shown in this video). But I am getting NEGATIVE values. This may be because the detection is not detecting the same objects on the left and right side at the same time!
Any helpful suggestions??
can you tell me the packages and algorithms to do that?
@@granatapfel6661 yolov8
Very informative video..i have a question what is the maximum baseline which can be set for a stereo camera setup???
I would really appreciate if u answer this 😊
Thank you! The baseline pretty much depends on ur application, if the cameras are too far away from each other u can only see very little in both images. If they are too far away from each other, u might not be able to see any correspondences
@@NicolaiAI thank you so much for your reply.
If the stereo setup is convergent axis setup( toed in camera setup) then what changes are necessary to calibrate the cameras to use them for depth estimation?
how about relative speed?
Does it also work with two sepatate cameras aligned
Yes u can make it work. I actually use that in the video
Hey man I am following this video series it is amazing work, I appreciate. I need to make a project which needs to recognaze the person and compare with the picture which we have already, after that the program should be say this person who shows his or her face to webcam is this person which we have the picture. Can I make it with opencv and this video series?
Thank you for dropping by! Yes u can definitely create something like that by using the techniques throughout this tutorial with opencv and computer vision methods. U won't get as good results as if u had multiple images of that person and could use AI. But it is definitely possible to do something with OpenCV! :)
Okay thank you!
Hi, I am facing error while calibrating imge
File "C:/pythoncode/StereoVisionDepthEstimation/stereo_calibration.py", line 75, in
retL, cameraMatrixL, distL, rvecsL, tvecsL = cv.calibrateCamera(objpoints, imgpointsL, frameSize, None, None)
cv2.error: OpenCV(4.5.5) D:\a\opencv-python\opencv-python\opencv\modules\calib3d\src\calibration.cpp:3694: error: (-215:Assertion failed) nimages > 0 in function 'cv::calibrateCameraRO
i have debug the code line by line and the image path is correct
debuged line: retL, cornersL = cv.findChessboardCorners(grayL, chessboardSize, None) it is not getting any corner and the retL is going false.
I am also having the same error. Did you got any solution to this error?
what camera u used ?
Just two budget webcams
How do I know the alpha value? Is it coming from my camera specifications? or you calculated it before?
Yes camera specifications. Its the field of view (FOV)
@@NicolaiAI Thanks! Just wanted to ask if my cameras are placed in orthogonal to each other can I use the same procedure to calculate the depth estimation? I guess then only parameter changing is the baseline b ?
It's a requirement that the two image planes are parallel for disparities to work. If they are not parallel the calculations are a bit more complex and the x disparity can't be used since the epipolar lines won't be parallel too
@@NicolaiAI Thanks. I got the idea now! I need to do rectification, two homographies (3*3 transform) and then epipolar line now horizontal :)
@@raihankhan5160 hello, I'm working about 2 cameras placed in orthogonal to each other too. May I have your contact email?
Can this project be done using a mobile camera and Arduino instead of a laptop?
Likely not as you need two camera and a mobile camera has one
I got problem on stereo calibration file;
on the line retL, cameraMatrixL, distL, rvecsL, tvecsL = cv.calibrateCamera(objpoints, imgpointsL, frameSize, None, None)
cv2.error: OpenCV(4.5.1) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-5rb_9df3\opencv\modules\calib3d\src\calibration.cpp:3694: error: (-215:Assertion failed) nimages > 0 in function 'cv::calibrateCameraRO'
Please can you tell me whats the problem?
Thanks a lot.
Yes that error is because it could not load in the images. Make sure the path is correct and try display the images when loading them in
@@NicolaiAI Thanks. Now its works!
Great!
@@myyard9693 Hey, can you show us how did you solve the problem ?
@@NicolaiAI Hey, can you show us how to solve the problem ?
I was able to calculate depth, but how can we calculate the width and height of an object ? I wish to calculate the volume. Thanks.
how can you able to calculate depth . tell me , btw i am already done by calculate the width and height of an object ,now i only want to calculate distance between object and camera .
how can you able to calculate depth . tell me , btw i am already done by calculate the width and height of an object ,now i only want to calculate distance between object and camera .
Hi Nicro. How to fix this error?
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
It’s not an error, it’s just info
@@NicolaiAI but i run this code and the camera is not open ;-; can u help me
Hey a small question. How do I know the Field of View of my phone camera?
That might be given some where in the specifications for ur phone on the internet what lens and so on they are using
@@NicolaiAI Appreciate the reply. Three more things :
(1) Is it necessary to have same focal length and FOV for both cameras?
(2) Where exactly are we using the focal length value (variable f) ?
(2) If I give recorded video as input, is it alright if I give same fov value, even if FOV is diff for both cameras?
U will need to have the same fov on both cameras or u will get really weird values
@@NicolaiAI Thanks. Really appreciate if you could answer 2nd and 3rd question too.
Third question won't work when no in 1 and second question is to do the depth calculation
Can i combine this with yolov7 and track the distance of multiple objects? If so, pls do a tutorial, would be a hot topic. Greetings from munich :-)
Hi I have a course about yolov7 then u can just run that on the stereo camera and use the same code as in this one here
@@NicolaiAI my Problem is the following. In Yolov3 and v5 it was easy to understand which line is the actual detector. In 7 not so much for me. So i just run it with the cmd detect.py command. Is there somewhere an indepth - video that explains what part of the code from v7 does what? So i can run it in Visual Studio Code or another editor for example?
Edit: Also ty for all your helpful videos. And how did you teach yourself those topics? Are there books you can reccommend on computer-vision ?
@@QuarktaschemitSenf in the course I just export it as onnx and also modify the detect file for live camera in python
@@NicolaiAI : So this is already part of a course you've uploaded? Or did i get you wrong ?
@@QuarktaschemitSenf yeah everything is in there already
I want to use the stero vision to get both the X and Y coordinates and not just the depth, so I made some changes to your code, I want to know if my changes are correct.
def find_depth_withXY(right_point, left_point, frame_right, frame_left, baseline, f, alpha):
# CONVERT FOCAL LENGTH f FROM [mm] TO [pixel]:
height_right, width_right, depth_right = frame_right.shape
height_left, width_left, depth_left = frame_left.shape
if width_right == width_left:
f_pixel = (width_right * 0.5) / np.tan(alpha * 0.5 * np.pi/180)
else:
print('Left and right camera frames do not have the same pixel width')
x_right = right_point[0]
x_left = left_point[0]
# CALCULATE THE DISPARITY:
disparity = x_left-x_right #Displacement between left and right frames [pixels]
# CALCULATE DEPTH z:
zDepth = (baseline*f_pixel)/disparity #Depth in [cm]
x_left = left_point[0]
x_right = right_point[0]
y_left = left_point[1]
y_right = right_point[1]
# CALCULATE THE DISPARITY:
disparity = abs(x_left - x_right) # Displacement between left and right frames [pixels]
# CALCULATE X and Y:
x = (zDepth * x_left) / f_pixel
y = (zDepth * y_left) / f_pixel
return [x, y, zDepth]
Also, it seen that the parameter 'f' have never been used in the project.
can i get github link for it
i didnot find the code plz share this code
github.com/niconielsen32/ComputerVision/tree/master/StereoVisionDepthEstimation
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
i get this and the program ends
^ I am having the same issue
Why do I feel like he's rapping 😅
Because I was a rapper before