Google's Enhance AI - Super Resolution Is Here! 🔍
ฝัง
- เผยแพร่เมื่อ 11 พ.ย. 2021
- ❤️ Check out Lambda here and sign up for their GPU Cloud: lambdalabs.com/papers
📝 The paper "Image Super-Resolution via Iterative Refinement " is available here:
iterative-refinement.github.io/
github.com/Janspiry/Image-Sup...
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Haro, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bryan Learn, Christian Ahlin, Eric Martel, Gordon Child, Ivo Galic, Jace O'Brien, Javier Bustamante, John Le, Jonas, Kenneth Davis, Klaus Busse, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Mark Oates, Michael Albrecht, Michael Tedder, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Rajarshi Nigam, Ramsey Elbasheer, Steef, Taras Bobrovytsky, Thomas Krcmar, Timothy Sum Hon Mun, Torsten Reil, Tybie Fitzhugh, Ueli Gallizzi.
If you wish to appear here or pick up other perks, click here: / twominutepapers
Thumbnail background design: Felícia Zsolnai-Fehér - felicia.hu
Meet and discuss your ideas with other Fellow Scholars on the Two Minute Papers Discord: discordapp.com/invite/hbcTJu2
Károly Zsolnai-Fehér's links:
Instagram: / twominutepapers
Twitter: / twominutepapers
Web: cg.tuwien.ac.at/~zsolnai/ - วิทยาศาสตร์และเทคโนโลยี
ah yes, the age of yelling "ENHANCE" at a dude on a computer is almost here
it will NEVER be here.
all any of these enhance algorithms are doing is making guesses and making up new information. it is super dangerous to treat these as anything but a novelty. one place where they could potentially be somewhat credible is DLSS where it is approximating a superresolution from the information in multiple consequtive frames, as that provides extra information an alone image could never have.
Good for entertainment, bad for important shit. Enhanced photos shouldn't be admissible as legal evidence for example
not in the court of law but in the vfx industry
@@DemonixTB maybe we can use this on security cameras and stuff. If we store in low resolution, we can store the footage for a longer and use this when we need the footage?
lol yeah i used to laugh and yell "unrealistic!"at movies when they had a blurry picture, and they're like "zoom in over there" and it zooms in and its all upscaled automatically, amazing that things we thought impossible actually can be.
What I've always wanted to do with super resolution is zoom in to some image, enhance it, then repeat the process over and over and see what kind of weird stuff happens
Atoms ( ͡° ͜ʖ ͡°)
You can try it it typically gets very weird very quickly.
It will probably turn everything thing into animals 😂
@@FreekHoekstra cool, all the more reason to try it
They have this tech in CSI:Miami, in the end it shows the home address of the criminal.
47.4% is even more impressive when you consider the maximum is 50% not 100% as, if you put two real images next to each other, you'd expect each to be equally likely to be picked.
I thought confusion rate meant 100% confusion was 50/50, as opposed to "chosen as real" percentage
if it reaches let's say 80% does that mean it's extra realistic?
@@eliteextremophile8895 If it reaches 80% it means it's creating images that people prefer for some reason, not that they are extra or more realistic than a photo. Could be that it's making people more beautiful or attractive in some way.
@@ZacDonald or maybe outside of aesthetically pleasing, it’s simply more believable. Or “more real”. Which is fascinating to think about as it puts the question of “what’s real/realistic” in visual representation again. Something that may be less questioned these days with the advent of high resolution media and “source-less” lighting in cinematography
@Caleb Strohl 🤯
3:40 I think it's worth to note here that 50% would be the perfect score. Because if people couldn't tell the difference at all, they would most likely choose randomly, resulting in them choosing the real one 50% of the time on average. So 47.4% is A LOT better than it sounds. Truly amazing. What a time to be alive indeed!
And to add the math on that, (50-33.7) / (50-47.4) = 6.3 times as good!
Good point. However, I think here the data has been adjusted to reflect this fact.
Sure, but at the end of the day the image is still fake. It can't reproduce information which is lost, so for example if there is a tiny mole or scar, that will be lost. It's a nice party trick though. And of course it can cause problems when this is abused and deployed to frame people or dehumanise people, imagine if some young kid is reproduced in a deep fake to show her doing something bad, such a kid could commit suicide. And would this be admissible in a court of law? It's party tricks.
Athenians experiment with a new weird political system called "Democracy". MEH. French Revolution. Boooring. American Revolution: WHO EVEN CARES. Electricity is discovered. AIGHT IMMA HEAD OUT. wait hold on..... ROBOT BRAIN MAKES BLURRY IMAGE LESS BLURRY!??!?!?! WHAT A TIME TO BE ALIVE AMARITE OR AMARITE
Yeah if you do binomial distribution getting 47.4% with 50% as the probability is roughly 0.65%
The previous paper is literally what inspired me to start learning programming and machine learning! I love this stuff.
Do you know where I can test out this super resolution on my images?
That'd be fun to put pixel art into or just anything to see what it created
Yeeeeeeeeeeeeeeeeess
this is like NVIDIAs DLSS methode ,not real Sharpening
@@darrennew8211 Corridor Digital is awesome
@@zombiekilla7463 What's real sharpening? Btw this is adding in detail not present in the input image, based on knowledge of human anatomy, light etc. Dors DLSS go that far? I'm guessing it understands objects and silhouettes to some extent?
@@darrennew8211 thank you so much!💗
These are some really crazy results. But I have on observation that might be interesting to some. I can easily tell which ones were upscaled as long as they were showing their teeth. I am a dental technician focused on facial esthetics and these upscaled images all have very bad/impossible tooth positions and esthetics. The problem with teeth is that they serve very specific and complex functions (chewing and phonetics). The upscaling AI of course does not compute the jaw movements and is not aware that their position affects both eating and speaking. A trained eye like mine can easily spot the errors in the first 10s of looking at the photo. There are issues like midline shift and inclination, incorrect gingival zenith heights, incorrect incisor lenghts, gummy smile, teeth not following the lipline ... Don't get me wrong these esthetic issues are present in real humans as well, but are definitely not that common as in every picture here.
This is just a constructive critisism and the results are still awesome and I am a believer that these issues will be fixed two papers down the line ;)
Next update will close their mouths so you can't tell real from fake.
Thanks for that input, very intersting to hear your POV
Just a note of caution: the AI essentially becomes an artist who fills in the gaps by imagination - the result shouldn't be interpreted as recovering reality from noisy data.
Yeah, we know
It really depends on how far the Ai imagination goes (must destroy humans etc.)
Yes, but apart from images used for courtroom proceedings, I doubt it's that big of a deal. Deep fakes are a much bigger deal.
I've been thinking for ages now that this sort of technique should be able to take older cartoons and de-interlace them and build up an image that's large enough for a modern TV. Aspect ratio would require either cropping or inventing details to the sides, but for just uprezing, that wouldn't be much of an issue.
Was already wondering how it would know the shape of teeth and pimpels etc..
Ai has no imagination, it's guessing the output from billion input images
CSI was just ahead of its time with the "zoom and enhance" technique.
It's more like zoom and imagine. It should never be used in criminology since neural nets just "make up" the details that are missing based on the statistical likelihood of the features in their original dataset.
Didn't Blade Runner do it first back in 1982 with the Esper machine?
Harrison Ford even says "Enhance"...
Except they used the result as a proof in a court, and not as it should be: an artificial reality that is maybe more beautiful but does not exist.
@@Alorand I suppose CSI took the idea from Blade Runner and enhanced it.
Enhance. Enhance again!
We need old 90s shows remastered with this
is there an executable for this? I want to try it out.
YEA IS THERE A WAY FOR US TO CHECK IT OUT
"Super resolution", but it appears this does faces only? If so, it's more like face synthesis, like NVidia have been doing for a while, and they've already done something similar with that model. Not that this doesn't have interesting applications. But it's not general super resolution, is it? It's more like the 64x64 pixels are input parameters - configuration for a face synthesizer.
Agreed! Why the example are just pictures of humans?
yeah looks like its making up completely new features that were not present in the original low res image, based on some trained neural network? looks impressive but its fake
ya, pimples out of nowhere for example. no way it could detect something with the size of a 1/100th pixel or so.
I really wanna play 1993 X-COM: UFO Defense in super resolution. I'm intrigued to find out what kind of artwork machine learning could generate while upscaling old games. I imagine one way to train it would be to down-scale high resolution footage of modern video games and use that as input and the original HD footage as the expected value.
Really looking forward to the day when this is applied to old PC games. Feels like there's so much potential to have AI "enhance" old grainy games and give them new life. Even modern games could benefit. So many amazing games from the early 90s that would likely work well.
for 2d games no, it would absolutely kill them
and for 3d it's more about the models than about the textures, creating a HQ texture pack for an old game is relatively easy (and there are lots of people who do it, actually), but when your model is 100 triangles, it will never look good no matter how good the textures are
@@geli95us Yes but the game is rendered to an image displayed to screen, so a similar imaging AI could be trained to account for this and make things look like they have more triangles.
@@JOSHZOL I don't think that would work, for various reasons, these AIs generate details that don't exist in the original image, to do that, they need a reference, mostly it's their training data, which is just the real world, if you try to apply that to a video game, the results would probably become eerly realistic, in a way that doesn't fit the game's art, if you say "just train it for games" it's not that simple, different games have different art styles, it wouldn't be easy creating an AI that worked well for all games, if you want to create a different AI for each game, you would end faster by manually creating the new models yourself (I'm not saying nothing can be done about this, but it's hard and there's little to be won, so it most certainly won't happen)
I.e., you want to see what Mario looks like in real life?
@@geli95us but you can train the ai precisely for such cases. then it is no longer tailored to the real world.
What's cooler is that there are at least like 5 major companies that I know of with their own versions of this that they're working on and improving.
Once this reliable this would be one hell of a compression method.
Imagine scaling down an HD image to 64x64 pixels (and then compress those as well). The amount of storage save would be gigantic
Well but it wont be a lossless way to compress and decompress it loses/changes some information which will bassically be almost useless as it changes data.
AI-based lossy compression would be really interesting
@@caveskeleton4674 True. But as long as results are still close enough I think it could be nice for some areas.
It will never be reliable though because it’s just an AI art.
@@lazypig93 True, but the question is how good it matches, maybe there would be a check if the AI is close enough before it gets compressed like that, maybe though there will be a use case where this isn't an issue.
"It can even deal with glasses" *Shows a man with hovering glasses.*
DEAL WITH IT
Denoising is only going to get better and rendering speed will reduce considerably
Fascinating. And kudos to the team for performing these initial test on a diverse sample of people!
Now I'm curious how this method can handle compressed images. As far as I can see all the examples where pixelated, but not compressed and I think compressed material may be a bit tougher than just pixelated image. On pixelated images data frequency is still proportional to the source material, but compression algorithms can change frequency of details, so my question is, how good this method is with irregular detail frequency on compressed images like heavily compressed, low resolution jpegs?
You could always pixelate the compressed image and run superresolution. Would probably yield similar results
I think in the future we'll stop using lossy compression and a new standardized file format would utilize deep learning super compression/decompression, circumventing the problem altogether.
@@emilcrafter Wouldn't that reduce the data in the picture making the results worse?
Read the paper
@@__Brandon__ larger dictionary, yes, but that one dictionary can be used to compress and decompress thousands of images, so i think it's worth the space
I’d love to see comparisons between original high resolution images and synthesized images based on compressed versions of the high resolution images. I’m curious how accurate the synthetics are.
The idea I can go through my old low res photos and "enhance" them to be high quality would be super cool. But, the more I think about it, the more I feel like I'm reading a BlackMirror script.
I think you are correct, this is indeed fiction. Their claims are impossible. And it's conveniently not available to anyone, but they do include paid ads and have a patreon.
Pretty cool I wanna see how old videos from the past will look like in the future imagine watching your favorite nostalgia on super resolution
What a time to be alive!
- a great and beloved man
You were diving a bit deeper back then into architecture and technical stuff. Hope you'll include more that type of info in your future videos as well, that was making them much more interesting
I've been developing a game for several years now, and realized at some point that the resolution of my assets was too low.
Using these AI resolution enhancement techniques is an amazing solution. I didn't even know it existed ontil recently, and when I used it for my own assets the result just blew me away.
Awesome stuff indeed.
I'm curious about doing this to my own game project. How do you train it for textures?
sweet, would love to see it done on some game sprite characters!
Can't wait to see vr games take advantage of this. So we can have near photorealistic graphics.
not exactly sure how this would apply to VR games. this method would be better applied to upresing older games
What a joy it'll be when State prosecutors get a hold of and misapply this technology.
Yeah, as with any tool, it's gonna be misused. This algorythm can only guess what's really there, so Let's hope people don't use it as evidence or anything of the kind, too much at least, we know people will use such things for getting fake evidence.
"Your Honor, this photo is unacceptable as evidence."
"Objection! I saw it work on an episode of CSI once!"
"Objection sustained."
I imagine (hope!) digital trickery like this will be as valid as polygraph results in the law's eyes.
They're already trying to do this. Some ai enhanced photos were just used as evidence in the rittenhouse case this week
It's still more reliable than human memory which is acceptable in the eye's of law
Wow! This is crazy! I'm curious how long this takes :P
You made me remember the projects of super-sampling textures using AI such as Chronocross & FF7 PC .
The Sony PS1 had poor resolution & frame rate that made it a pain !! especially if the development textures are lost.
this episode was blowing my mind the whole way through, i was sitting here asking "how? how is it so good?" over and over again. and people couldn't tell the difference 47% of the time? depending on the sample group size I'd say that's close enough to be pure chance. this is incredible.
Aren't downsampled inputs much easier to handle than those that started from a low res camera?
This will be huge for Mobile Phones, using Digital zoom. I mean, they already do that. But the level of quality of this is really impressive
amazing.. thanks for the video!
Can't wait for the consumer software to implement this, yes i have been using topaz giga pixel for years, but this new ai is a whole new thing. Really help for large printing business.
There are a few details at 2:14 that weren't available in the 64x64 image and had to be created based on learning from other humans (tooth shape, fine skin damage etc). Obviously that's a necessity since info is irretrievably lost in the downscale, but I'd love to see the ground truth image for that one, just out of curiosity. Also, i wonder how the arbitrary additions would be remembered so they can be added reliably to subsequent frames/scenes in a film. I'm guessing m the model of the characters identified would be refined during/before a first pass processing of the whole film? Bloody hard problems they're solving!
The 47.4% score is boggling. The max score is 50%, which would mean that a human being has about the same luck as a coinflip of getting the answer right.
Jupp! Really good! Took 45 mins for me to upscale a 64x64 px image. Like this, it'd probably take a few days for this to use on a reasonable resolution, let's say from an old VHS tape.
nice hope to use it one day for my photo bashing art.
Put in 16 bit characters and see Them in full HD 😅
I've been using Topaz Sharpen AI for the last several months. Topaz sharpening doesn't seem to utilize an iterative approach (at least it doesn't show it via the app), but it has performed miracles on many photos for me. The samples in this video look even better.
What I am mostly looking for is a tool that does this for video. I wonder if this kind of tool can be consistent across video frames? And how does it perform on non-human content?
Topaz has a product for video upscaling.
Would be interesting to see the temporal stability
you made my day better🌟
Extremely impressive. Nearly let go of my papers there.
Rockstar Games used an upscaler algorithm to interpret the grainy text in textures for their recent GTA Definitive Edition Trilogy, and people are spotting lots of misinterpretations. I bet SR3 would run cirlces around what they used (perhaps a good follow-up video).
this makes me want to try it out on old DOS era game character portraits
How long does it take to run? I couldn't find timing measurements in the paper. I know we saw footage of a game but did they do each frame separately and make a new video from them?
would like to see the doomguy's face from Doom "enhanced" :D
I used to work on super resolution on digital cameras. The thing to remember is that while these new results indeed look impressive, the algorithm is not maximizing accuracy, but rather believability. Sure, they look more believable but if you zoom in and look at the eye detail you can see the algorithm ultimately had to guess how the eye wrinkles looked like, which wasn't the same as the real image.
Other SR algorithms that aren't inference based need to take multiple images of the same object/scene and stitch/merge them together. This is arguably more expensive, but arguably more accurate
Yeah, that's what I thought. It can't do much if the information just isn't there. But for things where the original doesn't matter, just how good it looks like making old movies super high resolution or making video games look super real this is going to be a game changer in the coming years.
papers are flying like crazy!
The most fascinating thing about this is seeing the Oculus First Contact demo taking place in Unity with the default skybox.
Now we need this to become DLSS
I'd love to try this on my own since the results are just a too good to be true. At 2:19, did the AI guess the age of the woman and is it the right age? The pixelated 64x64 woman could be either 20 or 50 years old and I wonder why the algorithm chose 40 (approx). I really can't see any indication of tiny wrinkles or aged skin in the 64x64 image. I also wonder if the brown dot to the lower left of the mouth is really "real". It might be based on a single pixel with a slightly darker tone but I really can't tell how the algo figured that out. I'd really like to try it out and see what the limits are.
My guess is that you can extrapolate skin imperfections from the light falloff even in gradients that are only single digits pixels long. Youthful, less bumpy skin will have a distinct falloff profile that is consistent. Kind of like shading a ball with pixel art - if you create one with a perfect gradient from light to shadow and move around a couple pixels in the halftone zone in it's going to give the impression of texture. There are also anatomical tells like fat distribution and some very distinct wrinkles like the marionette lines that are visible even in the low resolution original. All of these things create consistent patterns.
Love how you used G Hinton's image
I'd love to use this on upscaling artwork.
Someone applies that to Riven's 640x480 images and we're good to go.
Hope it isn’t used by police to misidentify people from CCTV 😬
Agent: Arrest him, it's clearly him!
Researcher: You know this isnt quite what our net is capab...
Agent: The algorithm has spoken!
Imaging applying this technique to older films to get them to 4K or 8K resolution.
I should try this for old videos.
Someone should run this on the default Minecraft texture pack. I'd be curious to see what it would look like.
It’s been done. Check out some of this channel’s earlier videos showing this.
What a mind blowing paper, 2:18 I'm shocked that this method was able to recreate this woman's K9 teeth with only a few pixels
Can i ask wht going on,, is that they are using apps or what
Being able to use this on the Patterson footage alone would be amazing. And I do mean "foot"
I'm into drawing illustrations and if there is any connection to make it's that a computer learned by heard what faces look like!
I've had to do human super sampling you'd say, when for a comic I took a photo of my face, to base the illustration off of it, and it was blurry and dark. It wasn't easy, because I couldn't see where the edges of the eyes were. I had to use my knowledge of how eyes are and some trial and error. I bet that's what the machine does too?
How does this compare to Topaz Gigapixel AI? I would have loved to see them tested side by side.
I have Gigapixel AI, and it's pretty good, but it has a lot of "painterly" looking artifacts and generally comes out softer.
@@YostPeter Thanks for letting me know. Did you test it personally or are you speaking in general?
Topaz Gigapixel AI, is good upscaling general images (in which is very good), not faces exactly. This is more comparable to things like gfpgan or gpen (but this is more advanced).
@@bgtubber I have it and I use for mostly landscapes and portrait stuff.
Edit: Oh I see what you mean. No I didn't test on the images in the video, but the artifacts are pretty consistent on all types of photos, so I'm just assuming.
One thing I'm curious about image enhancement methods is if they are consistent when reversed.
Are the images that it outputs something that when reduced using a standard method to the input resolution gives an image extremely close or identical to the input?
I feel like a constraint like that would be healthy for increasing the quality of upscaling results.
I remember watching a video where someone criticized the "Enhance" shout on account of it being unrealistic... now we have this
I wonder if this can be used for gaming on low end laptops to enhance performance. (Render the game in a super low resolution and enhance it with the Ai afterwards). But it would only work if the process takes less processing power, than rendering it at a higher resolution... and if the process of enhancing is fast enough.
Unfortunately this type of tech is already being used in courtrooms, most recently in the Rittenhouse trial, soon to be elsewhere no doubt. There needs to be AI experts in court to explain the major differences between good AI guesswork and literal truthful, (but perhaps blurry) imagery.
‘Looks real’ vs ‘is real’. Good point!
It looks remarkable, yes, but is it that good with images other than faces? The lack of other examples makes me think the answer is no. Or am I wrong?
It's that good with faces and natural images. If you feed it a 64x64 image of a goldfish it will upscale it as good as with the portraits shown in the video (afaik from reading the paper). However, if you feed it something synthetic/man-made like a skyscraper or car it will probably yield results that aren't as good as with faces.
TL;DR: It's good with "natural" images.
@@mikeklubnika Ok, thanks for clarifying. :) Hopefully they expand and improve it to cover more scenarios. I would pay good money for such software.
You're not wrong, but faces are the pinnacle of test images. Brains are incredibly good (and darn fussy) at evaluating faces. It's definitely starting at the hard end. Like learning to ride your first bike on a double-black-diamond downhill mountain race.
@@R.Daneel yeah that's a really good point
"lack of other examples"? Watch the video again.
This is so cool. I can imagine in the future an AI can make any videogame or VR world hyper realistic.
Great! But I would love to see the results on images that were low resolution from the get go, and not downsamples of perfectly lit portraits.
With carefully selected blank or blurry backgrounds. This is clearly a face-trained network, not a general solution.
Soon, the "zoom in on that" totally fake feature from TV shows and movies might be a real thing. That's crazy.
yeah, a decade ago we laughed at those TV shows :-D
Can it do video upscaling as well? In real time that is. I'm just wondering how impressive it is compared to upscaling techniques offered by nvidia or amd.
I am sure it will have trouble with temporal coherence. But two more papers down the line...
@@shApYT :D
entropy just took a punch to the guts
This really seems like science fiction. Im looking forward to uprezzing my old 3.5 MP images to hd!
That's amazing but also its kind questionable how do you get so much details out of 6 pixels? it's physically impossible! and ofc unless there's something I don't know yet.
Basically the AI knows what faces are supposed to look like from the training data and makes a guess that comes close to reality.
@@mar_sze Cool but how about skin dark spots skin rash etc for example old people.
@@fahadal-asmari6893 The AI has the same real world training date as your brain. If your brain can tell what the hi res image is supposed to look like, so can the AI. Little imperfections are extrapolated as an educated guess.
Yeah, I just performed a reverse image search and confirmed that one of the images that were supposedly generated by the AI, was actually just an existing hi-res photo and it looks exactly the same. If you did it for one, it stands to reason that you did it for the others. This is toxic content. I don't see blatantly lying to your audience and posing as educational or journalistic as a form of entertainment. Hopefully other people seeing this comment can give it a nudge so others can see it and know to avoid this scummy channel.
This could make zoom on next Pixels even better. It's already good but this could make the photo much better even if applied to the final photo.
Very impressive... Do we know if there is some kind of similar on going research targeted at 1- Reconstructing the Dynamic Range (and lost info) of a low DR image/footage ? 2- Fixing the wobble artefacts on shaky rolling shutter camera footage, which remain even after current classic processes of image stabilization have been applied ? Both these would be the direct following greatest improvement over an image after resolution improvement.
So we could have old dvd series to real HD after all. Nice.
I would like to see this done on very old films
What a time to be alive! :D
This is fascinating. I am really starting to wish I was doing something more like these research projects rather than business software. Can’t remember the last time I could describe something in my sphere as “fascinating”.
It is the primary reason I am going back to school.
finally.. I will able to see youtube thumbnails!
I like the idea that games could spend fewer resources on graphics, or send images online in lower resolution, and then just use an algorithm like this to make it look good.
Thanks for this video and the links! It’s amazing! Is there a way to have directly access to this real-time interface?
Those DOS games will look crazy
Cool that they used Geoff Hinton.
This is amazing! :D
Now we just need to "rotate around the vertical"
I need to dig out all the pixelated naughty pics from the 1990’s.
I would be very interested to see how it performs with fairly noisy or garbage inputs
Are there any books that talk about the mathematics behind these sort of algorithms? I would really appreciate it.
Geeez you put out Amazing Videos
I can't wait to see what will become of this in say, three papers. At the current rate of research, I wouldn't be surprised if this can run in real time and be just as good if not better, nor would I be surprised to see it ported to VR shortly after just like FSR was
What a time to be alive?! For super resolution images?
I'm curious to know how it would turn out if you were just to change a few pixels?
Also imagine having this as an engine to remaster your old games :o
What a time to be alive!
Insanely impressive!