How Autofocus Works - Computerphile
ฝัง
- เผยแพร่เมื่อ 29 มิ.ย. 2024
- Why are some cameras faster to focus than others? Dr Andy French explains the different approaches the computers inside cameras use to focus.
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com - ยานยนต์และพาหนะ
It never ceases to amaze me how autofocus can focus on a tiny scratch on an aircraft window rather than the scene outside.
That is simple, it is looking for sharp contrast differences, since things in the distance are generally softer, because the light scatters and diffracts through atmosphere and the crack is nice and close, so it will be the best to focus on. Also sometimes it works to prefocus in manual focus beyond the window (not on phone, heh), so it will much easily find decent contrast outside, rather than the scratch right in your face. Also helps to put that scratch mark closer than the minimal focusing distance of the system/lens.
@@Django45 You just ruined his amazement.
@@thesidj understanding is even more amazing!
from the video, it seems that if that's a presisting issue, it's probably because your camera uses active/sonar focus method.
My theory is just in case the crack gets bigger then you'll never miss that caught in camera moment even if it happens to be your last.
About 30 years ago, I met a guy who was doing post-grad research in the mathematics of blurred images. I remember (after a few beers) commenting some thing like "Oh, great, we are trying to get our pictures in focus and you are trying to get them blurry." All I can say is, sorry mate: great work...
Couple additional pointers: On sensor phase detection is also provided by other manufacturers than Canon, but with different approach. Canon uses those dual pixels, which are two separated diodes. Others use a mask that blocks different half of two adjecent pixels, providing either the "upper beam" or the "lower beam" that went through the lens. These masked pixels are then compensated computationally on the final image, so you cannot see them.
Also, the phase detection is fast, but not really that precise due to the error sources involved such as imperfections on the lenses like chromatic aberration and lens's autofocus motor calibration vs. the phase detection array position. Contrast detection is less error prone, and does not need the lens autofocus motor to be calibrated, as the focus is done on the final imaging device (the sensor).
The best on sensor autofocus uses both methods - it first figures out the distance and direction where to focus with phase detection, but leaves focus a tiny bit out of focus in that known direction. Then the contrast detection is used to find the strongest contrast from that point forward. The end result is fast and accurate (not as fast as phase detection only, but with modern calculation speeds, the difference is not big).
Thank you for your comment. Quite interesting and much appreciated.
Thanks! The video was heavily biased toward SLR cameras and didn't really appreciate the performance of smartphone focus. I wondered about mirrorless since I have a Sony a6400. This helps see the benefits of contrast instead of only using phase.
@@Ck87JF Sony implements focal-plane phase detectors, which are built into the sensors (not clear in which way it works, though), and it works with contrast AF in something marketed as "hybrid AF". Phase detection is not very robust when the lens is not perfect and in most cases has limited resolution, while contrast based AF methods are more accurate but work slower.
I would assume for mirrorless cameras that use their image sensor as the phase deteciton sensor, there's no calibration needed. At least they don't have that classical adjustment option like DSLRs did.
FYI: He is saying "phase detection", not face detection.
Isn't that how those Casio CZ synths make their sound?
@@U014B That's phase distortion aka phase synthesis
I was waiting for the feature recognition of faces, then it dawned on me. Scrolled down, pleased to find your comment on 1st phase, ah, place.
vaze detection. #BritshPeople
I was wondering what kind of face would be detected by just two pixels
Video is quite wobbly so I assume, next video will be about stabilization.
The wobble is adding in post for audience engagement purposes. Thank you for participating.
You know when in Michael Bay movies scientists talk about something interesting, but not too groundbreaking, and the camera man is having a stroke and the editing is faster than the shutter speed of the camera? It is too feel intense! Well, here minus the editing, but plus the lighting - suspiciously unflattering.
I was thinking it was just me... What a perfect video to talk about this subject... While the video is 'in focus', still compared to nearly every other video it is jittery as heck, in both the video of him explaining this and the video of the paper he's drawing on... Could've done to adjust the white balance too on the paper portions of the video it kind of whites out what he's drawing a bit... LOL Either he's had a bit too much coffee this day, or something's wrong with the auto stabilizer LOL
I started watching this while I was brushing my teeth. It felt weird when the video didn't stop shaking when I finished.
lol
Super-hyper-ultra-duper-action-inyourface-wobblecam-turbozoom Riley makes a video on Autofocus. Oh, the ironing.
This was covered in the video.
@@recklessroges Uhh, no it wasn't. o.o
1:26 I once built my own active system by fitting a bazooka to my camera. It worked quite well because after it had, _”fired something out from the camera”,_ whatever was left of the original subject was far enough away I could just set focus to infinity.
works every time :-)
A photographer that doesn't use a Mac, I appreciate that
Man, PhotonicInduction has really changed a lot over the years.
sad react
Haha! Small World. Do you remember when he had a competition to win a 60Watt HID torch?
I'm the guy that won it! Poor Andy had to send it all the way to Australia! 😅
I miss him and hope he's doing alright.
Love photonicinduction
@@subliminalvibes me too. His last video with the home office blocking him bringing his new missus into the UK and the free energy device got a lot people worried.
Brilliant! Something I always wondered. Detailed and clear
Very clear explanation, interesting and informative, thank you!
That's pretty interesting! I remember an older film camera having a circular focus assist in the middle, IIRC it could use it to autofocus, but you could also use it for manual focus - it was seemingly showing different focus levels on each half of the circle, and you'd know you're in focus when you can't distinguish one half from the other. I guess that was phase detection! It also reminds me a bit of how gamma adjustment is done on computer screens nowadays :-)
Yep, I've got two older film cameras that have that. It's called split-image focusing.
Mine also have the split-image circles surrounded by a ring of microprisms, which also help to focus.
It's nice how intuitive the split-image focusing is. Very easy when you've got something like a tree branch or straight edge to line up!
Totally, I literally was a kid back then and it was easy to use even without explanations!
In the older pre-SLR days, there was also split field range finders which I always found easier to use than focus field ranging.
3:57 You might want to turn down the exposure 1 or 2 Stops next time you film a sheet of paper with direct lighting on it
You made a very good video. Thank you for sharing your knowledge
Nice! I was always kind of curious how passive autofocus worked. The system I imagined is most similar to the phase detection setup he described.
Everything is a universe unto itself; something taken quite for granted is such a marvel of math and physics and art.
Great video and explanation. Thanks.
Excellent tutorial. Thank you so much.
Thank you for this video! As a photography amateur, I've been really interested it understanding the difference between phase detection and other focus mechanisms so I am really glad you chose this topic for a video. However I have to say I really didn't get how it works based on the diagrams you showed. Why does it need a separate sensor? Why is the sensor only one dimensional? How do the two images appear on the two sensors separately...?
It needs a separate focus sensor because when you look through the viewfinder, the image sensor doesn't get any light to work with. And the sensor is not one dimensional, but phase detect pixels are. Most autofocus sensors have both horizontal and vertical sets of pixels.
So it's one dimensional because there's no reason for it to be 2D, you'd just get similar patterns swept round in a circle. All you need is one distance, so 1D suffices.
And the diagram of the two 'images' is a bit of an abstraction, really those red and green lines are just two separate rays of light, and behaving as a wave, many rays average out to make the curves seen. You can then use chromatic aberration to determine which waves (top or bottom) have ended up in which places
@@natan9065 Yes, there is a reason for them to be 2D, and they are. They're called cross-type focus points.
A prism or semi-silvered mirror is used to split the incoming light between the phase detect sensor and the viewfinder.
There's a problem with this method. Because the PDAF sensor is separate, and isn't directly on the plane of the main image sensor, it must be precisely calibrated. The ideal is for light to travel exactly the same distance whether it is going to the PDAF sensor or the main image sensor, so that they both see exactly the same focus. Any tiny discrepancy between the two, will cause inaccurate focusing. In practice there will be a difference, but small enough to be compensated for by calibration. Expensive DSLRs allow this calibration to be updated by the user. Cheap ones don't, and as a consequence can often suffer from an inability to achieve tack-sharp focus.
Having the PDAF sensor separate is therefore not ideal. It's only necessary for traditional SLR-style cameras with an optical viewfinder, because no light is hitting the main image sensor when the viewfinder is in use. Mirrorless cameras (or SLRs when using live view) have no need to do this, because there is no optical viewfinder - light is always coming through to the main sensor. So, for those that use PDAF, they put the PDAF sensors on the main image sensor, either using the Dual Pixel system that he talked about here (which Canon has been using for many years now), or by simply replacing some of the regular pixels on the sensor with phase detect points. But there's a downside here too, in the latter case: the pixels being used for PDAF can't be used for capturing the image, so they have to be interpolated when the final image is constructed so as not to leave holes. In some cases the PDAF array even has wiring going across the sensor which can cause banding. But the advantages of PDAF are considered so worthwhile that every mirrorless camera manufacturer uses it (or a hybrid system of both PDAF and CDAF). Except one.
Panasonic use only CDAF, but their "DFD" system attempts to address some of its shortcomings by using a knowledge of the out-of-focus optical behaviour of their lenses to compute the distance and direction that the lens needs to move to acquire focus. This makes acquiring focus in Single AF very fast. However, it still suffers from a problem that can never be completely solved for CDAF, and that is Continuous AF. With CDAF, if your subject moves, you have no choice but to hunt around for that peak again. PDAF always knows where the peak is, so (in theory) it doesn't have to hunt.
Quite a long time ago, I had to write an autofocus program for a microscope stage. I believe the way I did it was to apply a Gaussian blur to the image, subtract the blurred image from the original image and then try to maximize that difference. Similar to this algorithm I guess, but didn't need to decide a direction for subtracting neighboring pixels.
Subtracting a gaussian blur is how "unsharp mask" sharpening method works. :)
The green pen was barely visible.
Wow very interesting video helpful to know how auto focus works and why it doesn't work aswell.
Brilliant. Answered my questions.
I was waiting for this for a long time.
Thanks for the video. Really well explained and provided an answer to something I never thought about. Quick question. Is focus peaking using contrast to determine the in focus edges? Seems that would be the case.
Excellent stuff.
ImageJ is a free software that is getting me through grad school right now
Neat... This is one of those questions, i needed answers so badly
But not badly enough to look it up yourself? It's not like Computerphile is the first one to offer an explanation.
FYI : Dual Pixel is EXTRA USEFUL in video cameras, since you always are in "liveview".
Very interesting video. I always wondered how does autofocus work and I completely understand now.
Your cameraman should get a tripod or improvise for better stability of the video. It is kinda distracting..
so if AF is going to better if there are edges, should I aim on edge to be inside the bounding box when using single point focus?
And that's the nice thing about SLT's. Phase Detection AF during video.
Quite often you want to discover the structure in your image, not just in X and Y, but also in Z (depth), then apply rules to choose where and how to focus. Here is where the hunting of contrast detection stops being a liability and becomes a benefit, and where the whole-scene processing of contrast detection gains power of the (generally) linear strips of phase detection.
Simple or naive contrast detection algorithms use differences within neighborhoods surrounding a central pixel, which is equivalent to passing an oriented bandpass filter across the image. Why not capture **all** the frequencies in the image? This can be done using a Fourier Transform, one common formulation of which is the FFT. Taking three quick FFTs at slightly different focus settings can reveal an immense amount of information that can then be processed to extract 3-dimensional geometry.
You not also get the static structure of the view, but you can also do things like object tracking without knowing what the object "is", relying on the operator to simply indicate a region of interest to track and keep in focus.
While FFTs are great for auto-focus, they can be a bit heavy for object detection and tracking. This is where MPEG4 algorithms such has H.265 help out. Even if you are capturing your image or video as "Raw" data, the autofocus/autotracking feature is often using the video compression subsystem in the background to do its job. Which is why many cameras offer "Raw+Compressed" options, since it's doing the work anyway, letting the user decide how best to use the available storage space.
Amazing video, can you do one on line drawing algorithms
This explains why a camera with low dynamic range struggled to focus in darker areas.
Thank you
FEW methods of doing autofocus? I can only think of one - do edge detect on the image and move the lens until the edge detect returns highest values (on the part of the image we want to focus). I'm curious to learn what the other methods are.
There are a number of cameras out there that support a form of phase detect on-sensor. Canon calls this Dual Pixel Auto Focus, for example.
Interesting, thanks. BTW, what's the watch?
Can you share the python code for the autofocus test? more over what is the IJ script you're calling? Im working with some young, super smart kids, part of what we are learning (togeather) is optics and remote sensors. Your program would provide a great learning tool for them/us
Could you not run 2 dimensional FFT on the image and try to max out the energy in high frequency bands? Could that would be a simple measure of contrast?
Interesting, this explains why the cannon 40d I used to shoot with would snap out of live view mode for a split second while auto-focusing
in a quick mode it does that, yes. It switches to the phase detect on a specified focus location, focuses and then goes to live view again. *thumbs up*
this video is lit AF
The phase detection sensor looks like a mini light field sensor
That's essentially what is, yes. A 230 pixel sparse light field sensor.
I'm really interested how dual pixel phase autofocus works.
The same. It's an image sensor modified to be able to do phase detection.
Panasonic is interested too.
@@MarceloLangame lol...
it is the same as normal phase detect....but instead of redirecting the light with a mirror, it works on the image sensor. instead of splitting the beam somewhere else, it uses 2 pixels to measure 2 points of beam and phase detect.... that is the simpleset I can explain it
Uh, he was able to visualize the contrast measure algorithm so easily by creating an image where I*[x,y]=I[x,y]-I[x-1,y]. So the generated I* image would represent the contrast between pixels and would be as brighter, as sharper the origin image is.
How incredible are this guy's free-hand diagrams?
FYI he's saying PHASE detection, not FACE. It took me a while to figure it out and confused me at first.
This is fascinating to me!
Which branch of science/CS is this studied in?
I’d look for “image processing,” but it won’t be just cameras. The field also does a lot in visual effects, robotic sensors, and all kinds of different things. All of those fields use the same algorithms, but for different things.
Could you attempt to explain how the new Samsung ultra phone has "108MP" from a 12MP(*9) Sensor? And how it "fits 9 pixels in 1" (really not sure if I phrased that right, but it makes me curious)
13:28 It's not totally accurate, very often it's even hard to know IF you're out of focus with CDAF. Then there is the issue of local maximums, since the example given shows a perfect curve, but a real scene will have multiple peaks. PDAF has another advantage in that it can continuously adjust the focus, but Contrast detection needs to evaluate the change in the overall image to decide when to trigger a new hunting phase (or rely on an external trigger, like the user.
Also, phase detection pixels is available on a lot of sensors now. Sony has them on on all their higher end sensors, pretty much. Haven't managed to use it however.
Canons dual pixel implementation doesn’t waste image area; all phase detect sensors are image sensors and vise versa. So for pixel peepers and very specific lighting scenarios, Canons approach should yield better results (in most real world scenarios you never see any visual flaws from Sony’s approach, but you could be really unlucky and get the most interesting detail on the spots where you have no image sensor and pixels are just a blur of the neighbor pixels). I think that’s what they meant by Canon DPAF not needing separate optics, there’s optical drawback to their solution.
Why aren't the end screen boxes clickable? Wanted to follow up with one of the videos but can't. There's no link in the description either.
Hasn't been months since youtube removed this feature?
What is the normal use of that paper with six-line staves on it?
8:25 Python 2? It's 2020...
He runs WIndows you shouldn't be wowed about python 2
He's using Fiji's built in Jython (2.5) interpreter
As if Windows is deprecated like Python 2.
He's using Surface. Far better than many other Windows machines out there.
This wouldn't be a computerphile video this year unless neural networks were being used for autofocus. Knowing Japan, it'll be 100 years before it ever makes it to DSLR's while phones will have it next year.
great
So basically Marques gets a car and reviews it trying to make as many Doug DeMuro puns as possible.
Active autofocus: "So you can use sound..."
oh, please please please, someone make a camera which uses sound which is a recorded shout of "THIS IS MY AUTOFOCUS PING!"
Why are the over-the-shoulder shots, in which the camera is clearly mounted to something, shaking constantly?
The mount is moving. People constantly move even if trying to stand perfectly still because they breath and stuff.
@@dstinnettmusic No, the over-the-shoulder view is fixed to something. It's just... vibrating.
The truly hand-held shot is the main interview shot (why a channel with over a million subscribers doesn't use a tripod is utterly beyond me).
With a single sample, does contrast detection know if it is out of focus?
cool video
I guess modern cameras have both phase detection and face detection then. Neat.
And how does it know that the red peak is before the green one? You could have explain it properly.
Aka the red peak is above the green one
Maybe if he had a green pen that actually worked on camera you'd be able to see what he was talking about!
Another thing about the phase detection, is that there will be less mechanical wear on the actual auto focus motor and gears, so the camera will have a longer lifetime (The less a moving part moves, the longer that part takes to eventually wear out and fail). But these days probably not as important, since most people appear to bin tech every 3-5 years.
I have been meaning to ask this from an informed person but couldn't figure out who to ask. I think I have come to the right place.
Is camera autofocus a hardware thing or software?
I mean, can a company improve the autofocus performance through firmware updates?
Sadly it depends on the camera - some cameras have hardware solutions and some software, many a hybrid of the two. So it's possible to improve autofocus via a firmware upgrade but your mileage may vary! -Sean
What really annoys me is when my camera tries to autofocus and it goes past the point where the image is in focus or it focuses on the background and i have to keep trying again. Autofocus doesn't always work. My camera doesn't have a manual focus option and i can't afford an expensive DSLR with manual focus.
My pre-video guess - checking the deviance of an image processed by someting like CV2's Canny function.
My logic with this is that the more blurry an image is - the less the 'borders' would be detected.
That's basically right for the second method, except that the Canny algorithm has a few extra steps to remove "weak" edges. As he mentioned you would use something like the Sobel filter in practice. This is the first step in the Canny algorithm.
@@JohnLadan Yup, just finished viewing the video.
So if an image is out of focus, there's an uncanny valley between the two peaks in the two sensors?
Next Video: "how exposure measuring works"
I've always had some ideas on how the logic work - specially on the blurriness comparison. After watching the video, i am glad i was right. it was kind of obvious on most phone/camera where the images always go in and out of focus then in focus again.
Very well explained for a layman like me...
what kind of text editor/ide is this?
Canon did come up with dual pixel autofocus but now it's available on Sony sensors like Samsung phones use as well as others
Are those Sony sensors or Samsung sensors?
@@ArumesYT Last I heard Samsung were using Sony sensors but it's possible they've since stopped
@@kendokaaa They also use their own sensors since 2003. So again, are the sensors you're talking about Sony or Samsung?
Another thing to remember is that other manufacturers only use DPAF on small (smartphone) sensors. The most probable explanation is that they have a licence from Canon (who patented it ofcourse) to use DPAF , but that the use is restricted to small sensors so Sony and Samsung can't use Canon's invention to compete with Canon directly.
Nikon had phase detection on image sensor before Canon in the Nikon 1 series. The sensor was made by Aptina.
It's Mac Lethal
@5:40 Any pointers to an explanation of cross correlation please?
integral( abs( F1(x)*F2(x+t) ) ), find such t that integral is minimal.
Hi you need contrast for phase detect to work as well otherwise no 'peaks' to measure :-D
Category: Autos & Vehicles ?!?!?
Dioptic manifold clear
Isn't that the same as PID?
Kinda ironic talking about camera and the cameraman overexposed the paper shots
12:57: "We've gone through every pixel in the image here." You've kinda missed the whole first row.
First and last rows an columns. It's to avoid "out of range" error.
@@TheNovaq He starts checking from the second column because he uses "x - 1" in his calculation, but he should check the first row, there's no "y - 1" there.
Try warp stabilizer next time.
WHY IS THE CAMERA THEME WHITE?
is it face detection or phase detection?
Phase
thx a lot!
when I first heard that in the video, I thought that they were talking about face detection. but I was thinking that what they were talking about was not limited to "face" only.
when I saw the graph of the continuous line with peak on it, I thought of phases I learnt in electromagnetism and it looked similar
anyway luckily I have asked and u confirmed my thought gratefully
@@kelvinluk9121 You are most welcome, friend.
Face detection = detecting faces using image recognition algorithms. Phase detection = detection out of focus phase. Both words sound the same and are used commonly in photography. I’m sure a lot of people confuse the two :)
This was an interesting sequel to Blair Witch project.
Blur Witch project
Did someone send this to Panasonic?
The ilustration at 1:06, rangefinder focus, is fundamentally wrong. To make it right, at least front and back plane should move with different speed. Aligned plane is the one where the focus is.
Elves do exist in real life! Cool!
is it possible to use AI to Autofocus I think its better , nice video btw
Lost angel I'm no expert, but I think AI is starting to get used to autofocus in high end phone cameras. DSLR cameras like the one they show traditionally don't have as much compute power as current high end smart phones and instead have highly specialised hardware and software built only for capturing images.
apropos of Valentine's day...
For me, Autofocus has an anti-bias towards taking close-up photos. I hate it when i get it in focus, press the camera button, then it decides to throw it out of focus, then snaps the picture for me. Thanks technology.
Have you tried back button focus out of interest instead of half push of the shutter button.
@@christopherbuckenhamphotog368 yes. I just use manual focus whenever i have the time.
Fusion Developer the reason that many cameras have this issue is that the focus throw for non macro lenses is not very large up close. If you have a marked lens, you’ll notice that the distance from close focus to 1 m is very small. This means that you have to be much more precise to bail focus. Cameras are also smart, and they commonly look for something far away even if they can focus up close in case you are shooting through a window. This is why many cameras will hunt, focus up close, and then hunt all the way to infinity.
@@Van-yo2xr oh okay. It's just annoying that i tap to focus, then when i go to take a picture, it changes focus so the entire picture is out of focus, but now i understand why. Thank you.
@@FusionDeveloper that's why I suggested back button focus set your focus with a different button then when you press the shutter it doesn't re focus.
Has anybody else noticed Andy looks a lot like Jonny Sweet?
Anyone else looking at the thumbnail thinking this is a Photonicinduction video? Where's me 'ammer? I ain't 'aving it!
is this just sort of his hobby on the side?
Not quite sure why you need your focusing to work in complete darkness, since there's a lack of image ;p
More of a video problem, where you don't want the focus to slowly dial in when the lights turn on.
What's with the overexposure?!
next video
@@marc_frank How autoexposure works?
@@killedbyLife 😂
I liked this video because
Video storage for a month isolated
What kind of filter applied before
What kind.of compression and
What kind of sample size
Hurry. The next frame is hete
I expected the topic to be HTML input element autofocus
Dear *Panasonic,* please watch this video. Thank you!
LOL Olympus too? I think.
best comment
Panasonic CDAF (called DFD) is both faster and more accurate than PDAF. When you are happy with S-AF, which probably most amateur photographers are. C-AF is unfortunately flawed, especially in video where you can see the constant wobbling.
Olympus has PDAF in the high end models (all E-M1 models, E-M5 starting with E-M5 Mark III), and the CDAF in their other cameras is way better than what most DSLRs provide (canons with DPAF and the most recent Nikon DSLRs which come with PDAF on-sensor in addition to the seperate PDAF sensor excluded).
Hi
Get a better green marker!