This AI Learned To See In The Dark! 👀
ฝัง
- เผยแพร่เมื่อ 28 พ.ค. 2018
- The paper "Learning to See in the Dark" and its source code is available here:
cchen156.web.engr.illinois.edu...
github.com/cchen156/Learning-...
Our Patreon page: / twominutepapers
We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
313V, Andrew Melnychuk, Brian Gilman, Christian Ahlin, Christoph Jadanowski, Dennis Abts, Emmanuel, Eric Haddad, Esa Turkulainen, Geronimo Moralez, Kjartan Olason, Lorin Atzberger, Malek Cellier, Marten Rauschenberg, Michael Albrecht, Michael Jensen, Nader Shakerin, Raul Araújo da Silva, Rob Rowe, Robin Graham, Ryan Monsurate, Shawn Azman, Steef, Steve Messina, Sunil Kim, Torsten Reil.
/ twominutepapers
One-time payment links are available below. Thank you very much for your generous support!
PayPal: www.paypal.me/TwoMinutePapers
Bitcoin: 13hhmJnLEzwXgmgJN7RB6bWVdT7WkrFAHh
Ethereum: 0x002BB163DfE89B7aD0712846F1a1E53ba6136b5A
LTC: LM8AUh5bGcNgzq6HaV1jeaJrFvmKxxgiXg
Thumbnail background image credit: pixabay.com/photo-2618462/
Splash screen/thumbnail design: Felícia Fehér - felicia.hu
Károly Zsolnai-Fehér's links:
Instagram: / twominutepapers
Twitter: / karoly_zsolnai
Web: cg.tuwien.ac.at/~zsolnai/
#NightSight #NightMode - วิทยาศาสตร์และเทคโนโลยี
Just as a thought, if this network could ever be made fast enough to create these images in real time we could have night vision goggles that let you see as if it was the middle of the day.
Charlie I could imagine that AR headsets like hololens could come with such features in the future
I wonder though what those scenes actually looked like to the human eye? I don't really know how bad most cameras are in low light, but I can imagine the human eye could be closer to the improved results than the initial pictures without any goggles.
There is for sure some decrease in quality between the real life scene and the picture being taken, naturally. But a lot of these pictures were taken at night, the difference would be literally night and day between the AI generated version and what a human would see.
By using FPGA, I think it can run in real time (almost).
Should be able to get an approximation quite quickly.
This will be handy when SkyNet is locating the last Humans huddled together in the dark
don't give it ideas
Skyence 🤣
I'm glad we're considering Skynet's feelings here, not enough people consider just how hard it is, or how hard it can be on a poor superintelligence, to completely eliminate their creator's species.
If we get killed by AI, it's because we deserve it.
That could be achieved with simple heat detection already.
This is insane... we will be living in the world where nothing is real but accurately guessed lol
RandomTechS@#T
And this is very near to the common idea how our brain works. There no such thing like real, everything is interpreted and (re)constructed.
That’s exactly what your brain has been doing with data from your eyes and ears all your life - faking and interpolating. We’re now seeing how fake everything is that we perceive.
This is actually a problem that has plagued AI in the past, what is the "correct" way to see
If u treat ur brain as a much more advanced version of such vision AI, then in a sense we're living in a world where nothing is real.
SPOILER: we've always lived in that world. It's just that before it all took place in our heads.
I was holding onto my papers, but I still got whiplash from this technique. The amount on information that can be correctly extrapolated from the samples is staggering!
"computer, enhance!!!" "computer, zoom the image"... I'm glad we can bring these things closer to reality
"Give me a 360-degree extrapolation from the reflection on his belt buckle"
@@Zoonofski lmao
Everyone gangsta till you zoom in and it says "send help"
Wonder if the same technique can be applied to audio. Have 2 mics, 1 high end mic close to the speaker for ground truth, and 1 cheap mic further away. Learn to produce better audio.
Brennan Cheung - it can. Google demonstrated sound isolating AI that can pick out one conversation in a crowd. Search TH-cam.
It could also be applied to make cheap headphones sound far better like studio ones. Figure out how cheap headphones distort the sound, so after the distortions, it will produce perfect sound.
@@NolePTR 👍
@@NolePTR The problem with that is that we could already do it with simpler programs, we just don't lol Nearly all speakers, amps, etc. have their frequency response graphed in their datasheets, all you'd need to do is overlap all of these for a headset, and then send it to a company who makes an eq app that can then adjust the volume for each frequency spectrum. No fancy ai training neccessary
@@xxportalxx. AI could still help remove some subtle noise and "fill-in" sound information that was probably not captured. Adjusting based on frequency spectrum can be a further mix/mastering on top of that
Those before-after comparisons are incredible. Considering how much color is lost in the shadows of a low-light image, I imagine there's a lot of recoloring going on to get the output images. It kind of reminds me of the black & white -> color paper from a while ago. Really amazing work.
I would imagine that to the computer the color isn't actually lost, considering unlike our eyes color cameras only record color intensity. All it has to do is add to the color values it has, and tune them I'd imagine, the real magic is getting the weights and the sharp edges right
At 2:40, the window frame gets a distinct blue hue post-processing: that suggests its extrapolating from a few pixels of more blue data, rather than actually getting the colour right. As XPORTALXx says, its more about working out where the edges are, and knowing which artefacts are meant to be smooth and which have “texture”. Still, very impressive results, albeit an odd way to publish results (on TH-cam, rather than journals, but hey, welcome to this millennium I suppose).
@@andyowens5494 This video was not created by the original researcher. The paper was published traditionally, you are just watching the abstract
And bam one year later, my google pixel 3 take photo without flash in full night like it is daytime !!!
Night Sight is great, but it is mostly using high ISO and long exposures, and to a lesser degree image processing. At least nothing nearly as sophisticated as what is shown in the video. Note that the software you are using is available since the fall of 2018.
@@renerpho It is obviously stacking frames to increase the light received, for sure.
I don't know if you own one, but i was very surprised by the results, which are certainly on par with what we can se in that video.
"Note that the software you are using is available since the fall of 2018."
What's your point here?
@@ChaotikmindSrc I do not own one, but I have used one. Its Night Sight feature is quite powerful, and it gives good results in many situations. There is no doubt about that!
It does not solve the issue of increased noise though. In dark scenes, it still applies very high ISO to increase its sensitivity, which is different from what the algorithm in the video does. As a result, Night Sight's images can get quite noisy.
"Note that the software you are using is available since the fall of 2018."
My point is that the software was released too early to have the algorithm from the video implemented. Your original comment about "one year later" seemed to imply that Google had one year to get this included, which is not the case. Maybe that's not what you wanted to say.
Google has so much experience with AI that I wouldn't be surprised to see something like this in their next version of the software/phone.
It's actually like using the pro mode with long exposure and low iso ,
My budget phone camera can take better night shots than pixel 3 with pro mode 100 ISO and 32 sec exposure ,
I think I'll rather wait 32 seconds to take a nice night shot than spend 4 times more to get same result in less time
@@renerpho No dude there is no increased noise. I think it does apply some version of this algorithm to enhance the image. You must have used an older version of the Google Camera or something.
Thank you for making these videos. Its easier to keep in touch with the changing ML stuffs with simplified videos like yours. Cheers!
I *always* thought about this, even if the data is limited in low light why can't you just use multiple shots to work out what the image "probably" should look like? I've been waiting for this to happen ever since I bought my first phone that boasted good low light picture quality but ended up being total trash. like you said I hope this stuff gets put onto phones soon.
I wonder if you could process it on the phone, or if it would send to the cloud to process there.
omg, ok.. what are you doing here? :D
@@Xeller_real he does not need an entry ticket my boy.
@@marioisawesome8218 ok my boy
@@Xeller_real no i am the king you can be zelda
I was simply unable to hold on to my paper this time
I think there's no other channel on TH-cam that blows my freakin mind with each video. Wow...
I assume it takes raw sensor data, so there's a lot of information in the dark images we can't see. Even with Photoshop you can boost the exposure of the raw file after it's taken because the darker shades aren't compressed together, so this is mainly a denoising algorithm. Mentioning it because some people seem to think it extrapolates what objects the bright images would contain, or what shape or colour they are.
For me that's the crazy part- the necessary data and equipment for insanely amazing low-light camera sensitivity WAS ALREADY THERE, but we feeble humans couldn't leverage it.
@Adam Rath
Your missing the point we knew that the data existed. However instead of amplifying and guessing with educated math out way to how the image would look with more light we just expose our sensor for a longer time. Thus leaving the image 100% represented to how it should look in low light conditions. This does tho bring a new possibility that is taking low light images without potential motion blur.
@@IIStaffyII 👍
Exactly, most of the color info would probably be already there, it just has to tune the weights and clean the edges with some sort of anti-aliasing like program
@@xxportalxx. All the AI is doing is determining what is noise and what is not and then removing the noise.
Enhance. Enhance. Enhance.
Kids of tomorrow will be wondering what flash is.
👍
Kids of tomorrow wont be able to get creative either :(. “Seeing” the image is not the same as seeing the beauty; back lighting, side flash etc.
I think you are suffering from a misapprehension. No kind of data processing can ever replace lighting. Just like the bat needs to shout in order to get information about it's surroundings, our surroundings need to be lit up for us to see. And the better they are lit the more information there is.
@Taxtro its right since there is no such thing as completely dark neighbourhood . Its just that camera tech cant see but now with AI we can improve it to a greater length .
@Taxtro Ah ah ah, you're pulling a bill gates "We'll never need more than 400kb of ram" or something like that. AI has already demonstrated the ability to understand lighting, check out KaryKH's video on generating celebrities. The AI can understand light direction and shading to a pretty substantial degree. Who's to say we *won't* have digitized lighting in the future?
"Hold on to your papers" 10/10
"I LOVE IT"
As a photographer, I always wondered why it was that we couldn't have an algorithm that would do just this! Shut me up. LOL!! This is insane! I am sure they will also teach the AI how to keep the image quality post lightening! Can't wait for this update to my phone in the future!
"And now, hold down to your papers" - educational and entertaining, the best way to learn :)
thank you!
Now it makes sense why Pixel phones have such great image quality, even with fewer lenses or smaller sensor sizes than the competition. I didn't realise how much of a difference a little ML could make.
I wonder if you could synthesize the dark images, to get a much larger dataset? You could get millions of creative commons / public domain images and just make them darker and add tons of noise.
If the characteristics of the noise match the camera's noise patterns, I can imagine that. This is not an easy problem. There are still other roadblocks though, the darkening also has to match, etc. Cool idea. :)
the networks are also probably best trained for each particular camera specifically for real-time use cases. I think the training data is not hard to obtain anyway with a camera rig that is an array of cameras.
Interesting, because you have ML "painting" a better photo based on trained recognition within the noise. It makes you wonder how to define "real" vs "fake" photos, because it takes a lossy picture and adds detail.
What does this mean for evidence... At what point could "dreamed up artifacts" start to pop up? I think that's an important question, as these techniques might be used on CCTVs to solve crimes.
make darkening AI to train undarkening AI
Orian de Wit zoom and enhance!
It's AMAZING!! Thanks a lot for posting it!
This paper is awesome. It's really astonishing.
And also, look at the background in some photos: the trees look kind of weird 😛. Nothing surprising since they were not even distinguishable on the dark photo, but still fun to see !
This will be great for all of those security lights in cities. Can turn them off and still get crystal clear image of the perp... although light still functions as a deterrent and allows for guards etc to easily spot someone trespassing irl. But we should see some level of light pollution reduction which is much needed
Man, love your channel!
Great coverage as usual. This will certainly be used in future iterations of phone camera software. That said, the power of long exposure photography is vastly underappreciated here. Having a sensor exposed for long periods creates spectacular results, it doesn't just "maybe help a tiny bit". Also, long exposure shots are made out of actual captured signals rather than extrapolated data. The downside is obviously that they do require the phone to remain stable for a few seconds.
I always get serotonin bursts from your Ai videos.
This is amazing! Great work!
And as soon as we can get the processing time down to milliseconds using something portable, we have perfect night vision.
great video as always! really hope more people see this
Holly smokes!!! Always ready to hold on to my papers!
:-D
Your videos are always so delightful, thank you for all!
This is freaking incredible! Can't wait to see this in video editing software or blender or similar programs o:
I love it when he calls me a scholar.
You know it's gonna be a crazy one when he goes "Hold onto your papers"
*This channel is a treasure that we must protect* ❤️
Next up: an AI was found looking at humans while they sleep.
AI learned to walk, now AI can improve low light photography..this stuff is super amazing
For some reason my gut feeling tells me that with a reference image (black one) you can denoise like that without any AI. Not to mention that many neural networks of today shouldn't be described with words like 'intelligence' (as was shown with wolf/husky fiasco or those self-driving terminators pretending to be cars).
Simply brilliant! 👏
I wonder if their dataset consisted of pictures taken from just one camera. If so, then it is quite possible that they all have a specific noise pattern, unique to this model or even to this specific camera, so the trained network may not generalize.
Even if that's true, the result is still useful, since cameras could be in principle "calibrated" in the factory, with neural model fixing noise trained specifically for each one.
akrasuski1 Maybe but according to the onscreen text each image was taken with a different camera.
Generalization isn't that necessary here. If you are a camera manufacturer, you can train and customize the model only for your camera hardware and upscale the image.
They mention that in the paper at the end as an area of research for the future, as they currently assume it's trained for a given sensor.
Trained for a specific sensor is actually best. Quicker training and the ability to keep an edge on competition.
Judging by the researcher's video, they practised with several camera models, so it seems to be a pretty general NN
th-cam.com/video/qWKUFK7MWvg/w-d-xo.html
thanks to you i had an opportunity to learn about U-net(the architecture used in the paper). a nice paper. thanks!
That was breathtaking!! super duper cool boss
Impressive, as always !
This is amazing... I need this!
Great work! As neural network is not learning at run time, i.e. it is already trained using samples, you can convert neural network into a mathematical transform matrix.
Spacial domain transform are common practice. Using weights and activation functions, a special domain transform can be defined.
This will allow other mathematicians to further develop techniques for image processing, including frequency domain.
Incredible stuff.
Very interesting topic!
Amazing !!! great time to be alive !!!
I want this in glasses so you could see in the dark
This is amazing, I will try to see if I can train a neural net with old and remastered startrek to apply it to old episodes I'd like to see remastered too.
Truly amazing
Now we won't even be able to hide in dark when ai comes to get us
Amazing. Also very useful in real-life applications.
This is what we REALLY need in our phones and not 4 cameras or other shit
I wonder why they didn't just take long-exposure photographs and decrease the light intensity of them and add noise to create short-exposure photographs when they created the dataset? Seems like that would have saved them a lot of work, as well as the risk of accidentally slightly changing the camera angle or the scene itself between the photographs.
Appart from that, this is a really interesting application of image denoising. And the results are just astounding!
I have to say, the photo at 2:45 looks really cool with ISO grain
Amazing!
That is really really awesome
With the Note 10, I took a photo during late afternoon with high ISO and tweaked the white balance then showed to my colleagues. They were a bit puzzled because it was a daytime photo, but that looked exactly as if I took it few minutes ago.
If you do not fear to explore "pro" settings, you can already do fabulous things!
Even a picture of the floor with low ISO and big sun can look very good!
And with that kind of AI, well, it's freedom. Maybe "bye-bye" golden hour ^^
incredible! fantastic! extraordinary!
At 2:08 he uses the word "aperture" where he actually means shutter speed. A camera or lens's aperture is always open. It is the shutter that opens and closes. If we're talking about long exposures, which is he, he means a longer shutter speed in order to brighten the image or increase the exposure of the image.
Just for clarification, the AI-based system produced those images through convolutional neural networks without the use of flash?
Wow!
?
At first it blows my mind but as I think more about how a human eye can visualize things in lowest of the lights, I began to appreciate how closer we are in mimicking our ownselves.
Btw human eye resolution ~576 MP and color spectrum of ~7 Million colors
>Beating the top tier machine
The winning move is running away.
This is amazing
The uses for this lends itself immediately to the imagination. I hope we see this implemented in phones and editing software soon. Amazing
Amazing but scary at the same time
Very interesting video and absolutely to the point
*Holding onto my paper intensify*
For smartphone, you can use Google Pixel 2, 3, or 4, you can use feature on its camera aplication calls "night sight".
Mind Blown!
Holy smokes! I'm glad I was holding onto my papers!
So with GPT3, + GANs generating avatars + voice simulation from seonds of audio + full body deepfakes from a photo... reality is going to be an even more slippery business than it always was... yikes!
2 papers away from utter mayhem ... what a time to be alive... :)
Love the channel, by the way - keep it up!
Ganz außerordentlich. Den allergrößten Respekt.
Wow. That's remarkable! It reminds me another recent paper which denoised outputs from a global illumination image synthesis algorithm. If it works there it's quite likely to work also for photos.
For a layman this is a good example of what AI can do.
This is mind blowing
the title reads like a post apocalyptic journal entry
I really love your name! It’s like speaking a magic spell. Where are you from?
That's actually pretty amazing.
Thanks for watching and for your generous support and I'll see you...
in low light, durr!
wow nice guys !!
Infrared and thermal cameras: *_am I a joke to you?_*
Oh This AI thing ! Nothing seems impossible. I love this one.
It's just amazing
That’s amazing
If the performance of this is made fast enough and resource utilization small enough, maybe this can do live video too, on the phone. Oh man, I can only hope.
Heck even if it reduces the FPS quality in the live video feed, it might still be amazing.
he said it currently is 1 sec per picture, so either you'll have 25 cores and a 1 second lag, or use a special algorithm using the fact that videos are continuos
The fact videos are continuous can be used, definitely. There are compression algorithms that can handle this sort of thing.
This is really a WOW tech!!!
Freaking beautiful!
amazing. i must learn how to do this.
Wow this is an amazing job done with Neural Networks, I would have it in my smartphone hahah! Nice video!
This is just like movie VFX breakdown
Astonishing
Cant believe it's not even removing shadows to do this. Really impressive
but we DID get a smartphone implementation of this! it's in the new Pixel 4! I love it when you review an AI and it actually gets implemented somewhere!
Now I'll be waiting for this update on my Pixel 2 XL.
Well smartphones have some insane night photography now.
Was blown away how usable a photo was from an environment where I myself could barely see anything. The amount of detail in darker scenes is astonishing. Was iPhone 11.
*THIS IS AMAZING. I AM AMAZED.* sorry caps
I wonder if it can perform well on images that aren't so dark? One would assume so, but for the few examples I looked at (takes forever to load) they are all very under-exposed and then far more exposed. In most cases, we usually want to preserve the brightness as we see it with our eyes, but without noise, not make the image much brighter. Of course that depends on the use case, when you actually want to see much clearer it can be great, such as for night-time surveillance, conservation efforts, and many more.
Hi there! So I'm wondering if the usable .exe version is available for use via a download link. Because it seems that recompiling it would take way harder efforts. I just want to try it out without having to face many errors when recompiling the source code.
Also yes I would also love to hear if it is possible to batch select multiple images to denoise it simultaneously without having the user to tediously select all the individual images
Interesting. Very interesting.
This may be useful for self driving cars which also have to see in the dark. Processing time only has to improve...
Has photographer I'm so hype on this. Need an ai for overexposed to for her one file