ไม่สามารถเล่นวิดีโอนี้
ขออภัยในความไม่สะดวก
Light-field Camera - Computerphile
ฝัง
- เผยแพร่เมื่อ 15 ส.ค. 2024
- Shoot first, focus later: How does a 'light field' camera work? We asked Image Analyst Dr Mike Pound.
The Bayer Filter: • Capturing Digital Imag...
The Problem with JPEG: • The Problem with JPEG ...
Finding the Edges (Sobel Operator): • Finding the Edges (Sob...
Captain Buzz: Smart Phone Pilot: • Captain Buzz: Smartpho...
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscom...
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com
"…I looked through dozens of articles on how it works, and none of them told me." This made me lol.
Interesting. When I first heard of Lyttro cameras, I suggested this tech be used with eye-tracking so that a video would always focus on wherever the viewer is looking, thereby mimicking real vision and making the scene more immersive. I still think that would be awesome.
+IceMetalPunk Even better, use a light field display. It'd be like looking through a window.
+IceMetalPunk Head mounted ones do. But normal viewing LF displays have been made that are around a metre tall using hundreds of projectors. You can even buy small displays like LEIA 3D, though they call it a holographic display.
carrotSnack
Leia is not a light-field tech, it's a 3D tech. It doesn't adjust its focus at all, it just sends a left/right image to each eye. Light-field displays need many images from many angles, not just two. As for the meter-tall light field displays, do you have a link to one? I've only been able to find info about head-mounted ones.
+IceMetalPunk Dang, that sounds like a groundbreaking change for the movie media in the future.
Mitchell Gillespie
No no, you're misunderstanding. I'm not talking about focusing the *camera* based on where the *shooter* is looking. I'm talking about refocusing the existing video based on where the *viewer* is looking. As in, refocusing after the film has been taken, such as with light-field tech like this video describes, but choose the focal point based on where the viewer is currently looking with eye-tracking software. Both technologies have been around for some years now, but I've never seen them combined, and think it would be amazing if they were.
Very interesting way to refocus an image. Thanks for the video.
At about 1:00 he mentions that he'll talk about the more industrial variations of this technology "for a little bit". I'd be interested in seeing that footage if it exists :)
Coming soon! (needs a bit more editing!) >Sean
+Computerphile fantastic! looking forward to it :)
found "light-field microscopy": graphics.stanford.edu/papers/lfmicroscope/
+JohannesMP See the most recent video! It just came out.
So CSI can finally enhance photoes?
I wasn't ever really very interested in the light field cameras, seeing them as more of a gimmick, but seeing the tech behind them makes them incredibly interesting! Awesome video!
Dude's hair is on point.
+CatnamedMittens “Michael Bialas” Yeah, I'm jealous
***** 2fab4me
CatnamedMittens How are you everywhere?
I remember hearing about this years ago, I think it was at an MIT research lab. They were actually using a perforated piece of paper like a bunch of pinhole cameras instead of a micro lens, but essentially the same thing. I'm kinda surprised it took this long to come out with a commercial product.
This came out a few years ago
I want to see this technology being used in very strong macro lenses, where you have to deal with excessive DoF by doing focus stacking.
This tech could reduce such a photo down to only one image, and would make things a lot easier.
It has Macro from 0.0mm you can not have better macro than this on a regular camera so it is something
Can you put everything in focus at once?
+Holobrine Yes, you can. In fact, you can define a range of depth and objects within that range can be set to be in focus.
@@gfx2006 I imagine this would have to shift hard-edged boundaries, so you'd have to arbitrarily choose which ones to 'prioritize' would it not? (This could be done "automatically arbitrarily" I suppose.)
You guys should do a video on Li-Fi :)
+MrHappy i totally agree. Althou most of us already wiki'ed it i would like to hear from an actual professor
Very interessting - I hadn't heard of light field cameras before, and now I really want one!
Also, probably the best (shortest, most concise, most accurate, most memorable) explanation of why some things are in focus and others are not - thanks for that!
Lytro have something coming out that apparently captures every light ray in a room and using a VR headset you can look around like you were actually there in photographic quality and can move to look around corners
Then this image can be showed in a ligth field display, and you can focus the background or first plane with your owm eyes , just like a real live scene
Finally a nice explanation! Not that I was deeply researching the subject but normally I can get my head around of how stuff works, but this wasn't that intuitive...
Could you focus the entire image rather than just a region? I think that would be the coolest feature.
if it's like focus stacking what you mean, I can't see why not and with this same system.
YES you can. In real life you would need a tiny aperture (like a phone camera) to focus nearly everything, but you loose light. In the lytro software you adjust the effective aperture to a tiny value and acheive total focus without losing light. BUT in the norm of photography people think that big aperture looks better (more blurry foreground and backgrounds).
Idea: Attach one of these cameras to a drone that takes pictures of 3D objects and then try to reconstruct them in software, maybe even doing machine learning to teach a program to detect a certain object. You no longer have to worry about the machine itself focusing on the objects since the camera does captures all the data itself.
And you don't need two separate cameras to capture a 3d image.
wow they're using fractals for photography tech! This is awesome!
Ahh, that kind of reminds me of the grid mask used in front of color CRTs that allow each electron gun to only affect the subpixels for that specific channel!
Basicly the individual holes in that mask act like pinhole cameras, so that the way the thing is angled, the electron gun for the green channel can only reach the green phosphors on the screen!
Great video -- stable image, not shaky, yet cutting made it interesting without losing information. More like this!
@10:20 He was like "You know...optics :D"
ugh, proprietary software? I'm interested in getting one when someone makes an open source version.
+Dan Dart this could be something for you :) watch?v=p2w1DNkITI8
+Maik Kluwe thanks I didn't find anything quite there yet I could download and complete though
+Dan Dart I'm glad I'm not the only one who thinks so.
Definitely!
10:19 Fin the Human x2
How fascinating, I'd not even thought this kind of optical trickery would exist! :)
Nikon, Canon, Fuji, Sony.. watch and LEARN, because this is BRILLIANT!
The problem is that everyone wants higher resolution pictures and this technique actually lowers the resolution. That's the trade-off for 3D images.
We're getting one step closer to Deckard's camera from Blade Runnner. :)
Great video, explained something that has hurt my head since I first heard of it. Thank you.
This is fascinating. Would be a great device to be sold at a Sharper Image store.
Fantastic video! Finally understand how this technology works. Simpler than I thought - super clever however!
Wait a second, does this mean that the information in such a picture could be used to create a 3D model of the "depth"? I mean, it seems that the image contains depth information, effectively.
Yes that is possible but the effective dept is a proportion of the baseline which is very narrow. For comparison tv gesture recognition use about 15cm to recover dept at 1-5 meters in the living room.
+antiHUMANDesigns As Bogdan said, it is fairly limited in how much depth you can get from the image. The resolution isn't high enough to do something like that on such a small camera, you'd need a much larger setup for that sort of stuff.
One thing I would love to see is a setup where it uses 2 light-field cameras to create a scene. It would allow for far better resolution of the full light-field and can be used to create 3D models of a scene (of what can be seen, as usual. damn 3Dness of the universe)
Combined with some software, you could use it as a freeform 3D scanner that doesn't lose any focus as you scan the object in question. No more expensive rigs with motors, just hold it in your hand and get spinning.
Bogdan Manciu Yes, the information would be quite... low quality, I suspect. You'd probably get a rather jagged model, but it could probably be smoothed.
Kris Johnstone Yes, I agree, I was just speculating about the possibility of doing it with just one camera.
+Kris Johnstone
You could possibly tape multiple cameras together into a 5x5 square. It would be an intersting expeiment to see what result you'd get.
2:17 subliminal messaging? :P
+Bormeir Looked like some graph
+Bormeir Oh yeah - that's a proper glitch! - sorry about that - hope it's relevant to the video!! >Sean
+Bormeir Illuminati confirmed =)
+Bormeir I managed to pause it at the right frame. It's the figure from 4:41
+Bormeir
It's the graph that shows how an ordinary camera works, the one with the
I love these hardware videos, please do more!
Sounds like they took the idea behind the complex eye and developed this camera. This is some amazing technology. At first I thought it would just be some clever software gimmick that makes the camera take many photos in a short time but I'm delighted to see I'm wrong. If this technology gets integrated into VR applications, combined with eyeball tracking, depth of view would be much more realistic. A simple example would be Google streetview. The future looks bright.
If you have parallax information surely a 3D image can be produced?
thank you for the video. well explained.
could u use this to make 3d pictures?
Generally speaking you need 2 view points to construct the 3D pic, but this techniques seems to show you only need to vary the focus point to get the distance
Wow!
After weeks of watch and reading, I finally found a helpful Lytro video.
I have snapped a single picture yet because I wanted to understand the camera first.
Thank you.
I have both cameras.
Omg that old printer paper (4:19)! Who still has that? That's amazing! Oh yeah, and this tech is pretty cool too :D
11:24 explains why in this current form, this is not the future of photography... yet. For now it's completely closed and proprietary. We need an open standard for this and device agnostic tools (ideally open source ones).
+zaco21 Remind me again who owns the Nokia patents now?
+biohazara The problem, besides the patent issues, is the fact that it is still fairly low-resolution. That is more an issue with physics than anything else.
A higher-res system will require either considerably more dense CCDs or a larger CCD. (so, in turn, larger camera)
Larger camera will be a lot cheaper, which will then turn people off from it. (besides the really hardcore photographers, who would almost certainly welcome it)
The CCD density issue will mean a sharp increase in price since it isn't just that which is increasing in density, you are also going to need to increase the density of the microlens grid to improve its accuracy too.
Both of them work together to define the actual effective resolution that can be used for images.
+zaco21 also its not cheap, reminds me of FLIR cameras
+Kris Johnstone +zaco21 I completely agree this will take time to be something usable in the "real world". However, patents and the lack of open standards make progression even slower, much like 3D printing wasn't a thing until some of the patents expired.
+biohazara It will never be the future of photography. No matter what advances are made they are always going to be splitting a normal sensor up into a lot of tiny ones. That is always going to have terrible low light performance and it's always going to be very low resolution in comparison to a normal camera. I would assume it's going to be incapable of achieving the same FPS as a normal camera too considering how much more information it needs to process. It's impossible for it to ever catch up.
3:41 . OMG. I have not seen line printer paper like that for almost 20 years.
Clearly not watching our channel often enough 😉 >Sean
+Dexxter I've seen the paper in reams, it's a working line printer itself that I'd like to see.
+DFX2KX Same for me in a way. I have a carton of 2000 brand new 80 column punch cards in a sealed box, but I have not seen a working (or non-working) card reader since around 1990. I have been using the cards as notepaper for years after our university dumped all computer cards. When I got to the last box, I decided to keep it in case it became interesting for later generations. My kids will probably throw it in the recycle when I die. Oh well.
awesome video.. I only could comprehend without the math part though, but this is very well explained video and interesting tech
I know *everything* on the subject in the video but I wanted to give it a go...
Then, all of a sudden, 11:15 !!
And I was like "You wish!" X)
This whole explanation is purposely confusing and makes a burden of simple concepts which could have been described differently. I recognize that this isn't an "in depth" explanation but in +-12 minutes the whole algorithm can be shown easily and it's not "fairly complicated" but relatively simple if you have used it at least once in your life. :P
And if you happen to end up in a "black box" it's because you need to search for "integral imaging" to see some relevant results on the matter and you'll discover that "the black box" is actually public knowledge by more than a century. :)
How about using this tech together with 360 high-res video for VR experience application? You would basically look at different things inside your vr set and they would come into focus, making the whole thing much more believable. You would need to be able to precisely follow the eye movement within the vr set, but still, seems doable to me.
If you ditch the main lens and build a very large sensor panel, with some tweaking it could function as a 3D camera. You'd need a special monitor to display the result, but you wouldn't need 3D glasses.
beautifully explained!
The best explain on light filed camera.
Please help.. I don't get why you can't just reconstruct the image in the same way without the micro lenses by just keeping in mind that a certain amount of focus on the aperture will have light coming from specific sub-apertures go to specific places... He said something about how since it's not already focused, you have a lot of background noise or something... Let me know if you do.
every bigfoot hunter should have one of these
so why change the focus? Bring the whole picture into focus. Yes, it's a bit unnatural, because an eye could never do that (unless the iris was stopped down to a pinhole), but so what? Everything in view would be clear. Or would that be mathematically impossible?
+rchandraonline It should be possible, but not with the microlens array itself. It would take additional post-processes, along with edge detection to know when a part of the image was at peak sharpness while working through the various focal depths. So basically a program would start at one end of the refocus range, and rank each part of the image as to how sharp it is. Then step the the next depth, and rank it again. Do that for the full range, and then blend the maximally sharp areas together. There will be even more work required to deal with occlusion of a part of the scene by the out-of-focus blur of a nearer object. Though that may fall out automatically from the sharpness evaluation.
+Christopher Meadors I think that the Lytro software can do that easily. You can see at 11:45 on the macro picture of the fly, when he clicks on the fly's rear leg the whole the whole picture is focused for a really brief moment of time (it's better visible if you slow down the video).
Also the technique you're talking about is called focus bracketing or focus stacking and can be done even on classic cameras, but to do that you need to take multiple pictures with focus slightly changed on each of them. Then you can use a focus stacking software and process the pictures. You can achieve effects like a macro picture with perfectly sharp background (the effect is stunning, especially if the background is really at some distance from the object being photographed).
I've been playing with this technique myself few years ago with my Canon S5IS with CHDK firmware. I've run a Lua script on it that took a pre-set number of photos with focus bracketing :).
Wasn't this done in Citizen Kane?
Finally! Thanks for explaining this!
it does not neccesary make sence doing it that way
it also would be possible just to increase the sensor size and lens size so they make a bigger focus area
the lens in front the 3d lens also would have its limitations it certainaly has a bend point where it cant make a focus depending how big the lens really is and work
I think it's the future of photography. Nearest future.
+Imrooot At least for now the image quality is ridiculously bad compared to the traditional imaging. It's going to take a long while before professionals can start switching to these things.
It's not yet comercially viable. This is the future of mobile phone cameras that are 1mm thick and always in focus.
Alas the microlenses are hard to mass produce, but give it some time the material engineers will figure it out.
Bogdan Manciu The thing is really that I don't see out-of-focus pictures being such a big problem anyway nowadays. People are fully capable of using cameras with autofocus, so this technology is not really solving any pain points. It looks more like a "nice to have" kinda thing, probably nice if the image quality ever comes on par with the "normal" stuff, but not worth paying significantly more for.
+Imrooot Nearest and farthest future all in one image!
What file format is used for the live photos at 11:40 to 11:45? Is it a conventional format like jpeg but segmented in multiple rectangles somehow?
9:03 is the best explanation.
So it is the effect of lenticular lens hologramms turned backwards. oO I thought it takes multiple pictures in a short time while moving the ccd back and thus having a picture with different focus layers that you later compile to a single picture.
+Morkvonork You could do it that way but I get the sense that it'd be mathematically very difficult to figure out what was going on and reconstruct the rays. It'd also be quite complicated to move the CCD that quickly without changing its orientation at all. You'd be better off using a lens with a variable geometry to bend the light at a slightly different angle.
I have purchased a LYTRO, but I cannot find the software to view. The link sends me to Raytrix, but they said they do not support it...any suggestions where I can get a download.
can you talk about RAW processing?
also, i need a program that can make HDR... i have photoshop, but i don't have Lightroom...
+Adrian Freund Darktable is great! But I'm not sure it's available for windows...
+Justin Koenig To make an HDR I would recommend Luminance HDR and then photoshop/gimp/darktable to tweak it (Luminance HDR can take several types of RAW input!)
I'm sorry to hear that, it works pretty well for me, no exposure problem (you can set them manually though if you can find the values in another software). It even has some great batch processing capabilities if you want to create hundreds of HDRs at once. It is very cpu demanding so I do understand that it may not be for everyone. I would advise finding the right profile/settings at a low resolution and only when you're sure let your computer run overnight to produce the high res of the batch of photos. Also make sure to enable multithreading if your cpu has multiple cores.
+Justin Koenig You could trie Lightzone, played a bit with it some time ago. It seems to do what you want.
+Justin Koenig you can use photoshop to make a HDR...
I wounder how this would combined with photogrammetry.
So after 6 years we have computational photography on iPhone! How are they different from each other?
Shoot a video of a still frame.
Do a focus pull.
Watch the video and pause it whenever the desired distance is in focus.
Bim Bam
Boom
is that possible to extrapolate the shape of things out of this?
Could you reconstruct the image in 3d this way?
zolika1351 Absolutely.
What about cmos sensors? Are ccds the only sensors?
0:03 Which are what?
How much more space is needed to store this informations compare to a normal image of the same resolution?
+DynoosHD Theoretically the same, but in practice the files are larger. I don't know the exact details of their storage format, but while you'd imagine it's just the same size as a normal image (because it's still a regular CCD behind the microlens array) they can't do as much compression because that would risk affecting the image reconstruction. My lytro images were about 10-15 MB, and the newer illium camera is closer to 100 MB per image.
What if you made a large main lens and multiple layers of micro-lenses?
Could the micro lenses be replaced with simple appertures . i.e. cheap pin holes?
Yes, and the mathematical models often assume that they are!
could you link a large amount of these cameras together and create an image with a wider parallax ? kinda like creating another micro lens, or macro lens.
I wonder what file sizes are like.
so is it possible to focus on every part of the image at the same time?
I played around with images that other people shot, i think you can
the HTC One M8 does something similar (effect, not implementation), no? but with two cameras?
So in other words there are more pixels in this CCD than in a common CCD camera...?
It'd be more interesting to use a 50% random mask (instead of a front lens), and do a 'hologram-like' inverse function on the CCD output...
Snot's Ham was the origin of the name Nottingham apparently, thanks Jeopardy! haha
Ah so it's a camera taking 1000's of tiny pictures of the microlens array.
Cameraception!
Could a comparison be made between a human eye and a bug eye? (classic vs Lytro)
All very good, but really what's the point of it other than the gimmick of refocusing after the event?
Amazing
Second!
So the HTC One M8 has one of these? But it has 2 cameras so how does that work then?
Graham Smith
Any picture you take can be paralaxed like seen in the video by tilting your phone (the "3d" option) and any picture can be refocused afterwards pretty much identical to how it looks in this video. You can also upload it to an HTC website page and send that to a friend so they can refocus online. It's pretty amazing but I don't know if it's the same.
+Niels Schellekens I've just had a quick look to see if I could find how the M8 refocusing works, I didn't find a detailed description but I think my first reply was basically right. It seems the two cameras have quite different numbers of pixels. There's a main camera to take the photo and the second camera just captures depth information at a lower resolution. I wonder if they are having the second camera capture multiple shots at different focal points in order to gather the information necessary to correct the focus in the main image.
+Niels Schellekens The video shows parallax along two dimensions.
You can only do it along one, along the axis defined by the two lenses, right?
Major Gnuisance
It's tilting upwards and sideways if that's what you mean.
Graham Smith
Would be nice if someone knew how it worked. I'm by far a camera techy but I'm very much enjoying this series of Computerphile
What did he say @1:12 "i am holding up a lightro mainly because.."??
+Aryan Arora "I own one. Or let's say the school owns one." So he's holding this brand of camera up because he/the school has one.
+Aryan Arora "I am holding a lightro mainly because I own a lightro, or I should say, the school owns a lightro. Uhm...:
when we talk about micro lenses , would it mean there will be a micro lens for each pixel ?
+krishnan srinivasan No, consider te Lytro Illum (not the one shown), it has 200,000 micro lenses and a 40 Mpix sensor. The exported image is, howewer, just 4 megapixels. So the very concept of pixel is not that simple in a light field camera.
Thank you 😃
can you guys do a video on bokeh filters?
Could someone use one of these very expensive cameras to take an image of the jastrow duck rabbit "illusion", and see if shifting the point of focus to infront or behind the real image makes the image flip?
I suspect such may be the case because if one places a thick glass sheet over a printed copy of the duck rabbit, and then alternates focus point between the top and bottom surfaces of the glass the image can be flipped.
there's been a few pyschology phds which used binocular eye trackers, and which say that can't be an explaination, however basic trigonometry shows the eye trackers used just wouldn't have had the angular accuracy to determine depth of focus any closer than +/-5cm over about 30 cm viewing distance.
+steve cummins The retina appears to be non optimal, with light having to pass through nerve fibres etc before hitting the light detectors. Could those those layers be functioning as a micro lens array?
Can you put the whole image in focus at the same time?
If you own a Lytro, you can skip to 03:30 without missing anything.
Interesting, but camera can take images very fast nowadays, why wouldn't you just continuously take images at multiple focus levels? And if you wanted the parallax effect then move the lense also?
Because even very fast isn't fast enough when objects or the camera are in motion. You get motion blur. And combining the parallax effect with refocusing would be impossible as well.
why?
brilliant!
I love his accent. Where's he from?
***** Thanks.
"Optics, you know" Yup, this sums it up rather nicely.
But if I understood correctly this kind of camera requires very high pixel density CCD and gives rather low resolution image. Just like these "no glasses" 3D TVs.
I should bring this to school...
Mike looks like Wikus van der Merwe from district 9 in the thumbnail
Whoa, imagine what this technology could do e.g. with a fisheye lens. Or making 3d movies with a single lens.
I suscribed because he likes photography
Seems like you could even do away with the primary lens. This could be a leap for 3D cameras, and displays as well, by reversing the process with high-density led screens.
pratherat, you can do away with lenses altogether in some forms of imaging.
@@johnnyswatts I'm sure there is much beyond my knowledge. I remember from my toddler years (early 70s) I had a book with a lenticular grating on a specially printed image that gave it a 3-D look. These were vertically oriented, but I've heard of "fly eye" sheets that work in any direction. I've been unable to get my hands on these, although I have a stack of the vertical ones in my desk drawer.
My point was that the process is reversible. Image projection and reception are the same light paths (ideally). A properly shaped lens in direct contact with the image sensor chip should culminate parallel rays at a single point on the sensor. An array of these could produce a 3D image when displayed in a similar way. No primary lens needed.
Thank you for bringing this back to the forefront of my thinking for a while.
Soo... is that how a bugs eye works maybe? The array giving the bug's brain more data.
If you use micro lenses not in front of a sensor but in front of a display, isn't it than possible to create a 3D display.
9:03 if each micro lense is just a tiny camera taking a picture, with micro lenses in front of a display it's tiny screens showing different pictures. With that it should be possible to create a 3-dimensional picture for an observer.
Absolutely! There's lots of work being done in this area. If you're really interested, google "light field display".
would it be possible to focus every area of the image so the entire image is in focus if that makes sense, would just be wierd to see xd
Would this be useful for astronomical purposes?
Quantum Custodian I was thinking more more for the parallax
+2chws For photographical purposes, astronomical objects are infinitely far away. This means you don't need to refocus and you also don't have any parallax effects, at least not by moving the camera a few centimeters.
However, in radio astronomy a similar technique called beamforming is used to get a virtual directional antenna that you can redirect at different objects in post.
+koseq7 what it that .&.qu.ot; thing.?
Penny Lane wow
***** Yeah, the TH-cam comments have accumulated quite a few bugs in recent times. But this one should be so easy to fix that it's the ultimate prove that no-one in this company has ever come across it themselves, never mind some kind of QA department one could expect. As if a lot of the people here weren't bad enough, now we also have to fight the technology :/
I'm curious what would it look like if it was a real film rather than CCD.