Google’s New AI: Fly INTO Photos! 🐦
ฝัง
- เผยแพร่เมื่อ 10 ก.ย. 2022
- ❤️ Train a neural network and track your experiments with Weights & Biases here: wandb.me/paperintro
📝 The paper "Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image" is available here:
infinite-nature.github.io/
❤️ Watch these videos in early access on our Patreon page or join us here on TH-cam:
- / twominutepapers
- / @twominutepapers
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Balfanz, Alex Haro, Andrew Melnychuk, Benji Rabhan, Bryan Learn, B Shang, Christian Ahlin, Eric Martel, Geronimo Moralez, Gordon Child, Ivo Galic, Jace O'Brien, Jack Lukic, John Le, Jonas, Jonathan, Kenneth Davis, Klaus Busse, Kyle Davis, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Michael Albrecht, Michael Tedder, Nevin Spoljaric, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Rajarshi Nigam, Ramsey Elbasheer, Steef, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Timothy Sum Hon Mun, Torsten Reil, Tybie Fitzhugh, Ueli Gallizzi.
If you wish to appear here or pick up other perks, click here: / twominutepapers
Thumbnail background image credit: pixabay.com/images/id-1761292/
Thumbnail background design: Felícia Zsolnai-Fehér - felicia.hu
Károly Zsolnai-Fehér's links:
Instagram: / twominutepapers
Twitter: / twominutepapers
Web: cg.tuwien.ac.at/~zsolnai/ - วิทยาศาสตร์และเทคโนโลยี
![CAMPปลิ้น | EP.77[2/2] "รับทราบ" แก๊งเพื่อนรัก 10 ปี ที่ไปไหนก็มีตำนาน!](http://i.ytimg.com/vi/ifc0ofD51B0/mqdefault.jpg)
You do understand that your child like glee for tech is a beautiful thing, yes? I hope you never lose it.
You are very kind, thank you so much. I can't help but love these amazing papers! 🙌📜
@@googlelord1678 Hungarian accent :) but human indeed.
I noticed how I take the auto removal feature on Pixel phones for granted already. It's a pity that we get used to things so quickly :/
two minute papers ai conspiracy theory ?? i mean .. considering what ai can do these days ...
@@istvanmeszaros4112 Accent is one thing but his cadence and strangely dashed speech is weirding me out.
I absolutely LOVE that the old "zoom and enhance" movie meme everyone laughed at back in the day is reality now.
Yeah I never would have thought Deckard saying "enhance, enchance" in Blade Runner would be a thing we could actually do!
Well, technically we were right, it's not and will never be reality. You can get an AI to dream up new pixels, but it can never be used for forensics. E.g. if you have a low resolution image with a car in it, and want to read the license plate, you could ask the computer AI to "enhance" that license plate, but it will just "dream up" a random one. You'll never actually be able to recover information that simply wasn't captured at the time. So those memes are still very accurate, because that kind of forensic usage of "zoom and enhance" is just dumb.
@@itsbazyli we won't know until someone trains an AI to recreate the correct character sequence from a collection of 20 pixels.
if it can guess 99% of the right characters based on the pixels it might be usable
@@ayoCC one of the first applications of AI was exactly information recovery from blurry/low-res text in images (CAPTCHA), so we already know the answer to this. You're assuming the information is still there, just degraded. In some instances it is fact the case. But below a certain threshold, the ambiguity between letters/symbols is too high, because the pixel values are simply identical. And if you take into account any real-life scenario, you have to add in variability due to differing fonts, shadows, angles, dirt, reflections, camera noise, heat distortion, lens distortion, and many other factors. It's mathematically impossible to recover information once its been lost. If you're lucky, the best you recover at that point is a list of most likely hits. With the aforementioned license plate situation, it might be enough to make a list of most likely suspects, if they go through, say, 100 potential matches, and lookup those cars. But it still won't be "zoom and enhance", because that would require a close to 100% confidence in data recovery. In the real world, it might be "let's run this specialized license plate model on the photo to try and get the list of most likely matches to cross-reference with our car database".
@@itsbazyli well there is one instance where you can actually “enhance “. If you are trying to read a license plate from a video recording, then you have multiple images containing that plate. You can then combine them into one enhanced image with higher resolution!
I imagine someday devs will make games with just points of interest and level design then press a button written "Auto-complete-map" and the AI will generate the rest of 3D environment based on sketches
it'd be nice to be able to paint backgrounds then an ai generates the geometry, and low res pictures for grass on mountains etc...
Ugh, that can be terrible, as if open-world games aren't getting too big right now
Dolores in westworld season 4!!
My take is that anyone, especially good story tellers will be able to enter a VR space and being telling a story. An AI will use their descriptions of the scenes to generate content on the fly. Story conjuring. The story unfolds in visual 3D wonder as the words are spoken.
You won't need developers. Alexa just asks you a few questions about how you feel today and AI will construct the perfect game for your moment. The same with movies.
"Today I want to see a new movie with Arnold Schwarzenegger in his 40's." And so on, you get the picture.
Imagine this kind of technology being used on Google Maps, that would be so crazy!
Imagine if the 3d view could zoom in and fill in the missing detail kinda like microsoft flight sim. Maybe even making it fully dynamic with weather and day/night.
That's what I thought too, but if what's being shown is just an assumption of reality, then is it really useful?
I think you may be excited by Google Earth. They had a VR app where you could fly around the world - one of the coolest apps I've ever tried.
th-cam.com/video/0QRGrXt0aJs/w-d-xo.html
😓ITS FINALLY HERE😓
that was my first thought hahah
I wonder if this could be used to improve transitions between snapshots in google maps?
yea i really hope google could use this someday to create a sort of 3d world where you can just walk through! and maybe even create more 3d versions of places with Streetview, so when you look from satiltie you can see all sides of buildings with streetview.
sounds awesome, especially with more pixel data provided by 360 imaging
for sure! be sure to check out apple's version of street view too, their transitions are surprisingly smooth! can only imagine what it would look like with AI like this
It would be cool if they use that together with vr, so you can actually walk anywhere in the world
@@notuxnobux Yes! I thought walking in google maps vr was a option, but was dissapointed it wasn't but that would be so cool!
Dungeons and Dragons games of the future are going to be wild if Dungeon Masters can just narrate explorable worlds like this by just speaking into image generators like Stable Diffusion
Imagine GPT dialogue with npcs, and like a very art direct-able 3d generator. You could even give verbal input as a DM as the story goes on.
The new method reminds me a lot of how fuzzy everything seems after waking up and recalling a lucid dream when I go flying but not really focusing on the terrain, like if I look away from something and imagine it differently when I look back it's like how I imagined the change
2:40
Finally, the following line in StarTrek makes sense:
- “computer, magnify”
- “enhance image”
Hahahaha! Or the CSI scenes! It sounded so ridiculous back then!
@@ShuyaTheDark yes. They even parodied it in futurama. (Why is it still blurry? - it doesn’t work - it does in CSI miami)
Lemme find the clip :)
Here we go: th-cam.com/video/WwnI0RS6J5A/w-d-xo.html
I love how this is the kind of stuff I dreamed of as a kid. I always wondered why computers couldn't be used to do things like this. Turned out it was just limited by processing power and sophistication of algorithms. I spent some time recently making zoom-in/flythru videos with Disco Diffusion but it was extremely hard to tune, took anywhere from very long to incredibly long, it was nowhere near photorealism, and it almost always washed out into generic purple blobs after around 100 to 200 frames. Very impressive stuff here!
WHY DID HE UPLOADED THAT VIDEO IN 9 / 11 ????
????
@@aiisnice1453 You bring some funny things together that no one else would have noticed.
@@MrGTAmodsgerman WHY DID HE UPLOADED THAT VIDEO IN 9 / 11 ????
???? ??
@@aiisnice1453 it's been 21 years, my guy...
If every tragedy prevented anything on its anniversary nothing would be done.
th-cam.com/video/0QRGrXt0aJs/w-d-xo.html
😓ITS FINALLY HERE😓
This _really_ reminds me of the sort of imagery that my brain generates when I’m running or flying in a dream, the way details morph into each other in a way that’s… not really all that consistent, but does provide a fluid stream of imagery.
Same
Truly incredible paper.
The progress the last few years that I followed your channel has been unbelievable. As you always say, what a time to be alive. 😁
Yas, can't wait to play the next version of MS flight simulator. I can finally see my house in detail!
Yes. And then you see yourself through the window, playing the game😂
WHY DID HE CHOOSE THAT DATE
@@aiisnice1453 Beauty is in the AI of the beholder
Someone should make an IA that looks at the input and where one paper landed, then shows the results two more papers down the line. Now we apply that AI to itself and we've broken AI research. Thanks for coming to my TED talk
It feels like in the next 5 to 10 years humans are going to have a really hard time knowing what's true in the media. This tech, in maturity, is like WMD level power. Incredible to be witnessing this era.
Also everyone will be unemployed
Most people are already struggling with that.
You cant trust media now, Parly due to bigtech like Google
A good fiction isn't inferior to reality, and one leaks into another. In the end days, there'll be just you and your Matrix. "There is no spoon." Base reality is so last decade.
you sure you didnt write this commemt in 2000?
3:00 I'd love to see this used on old CDI and laser disk games (to see the difference)
Right?! I want to put this on Perfect Dark or Turok lol
I feel this is 2-3 more papers from being amazing. I'd love to also see AI find pictures taken close to the same area to make the generation more accurate.
As commented below the previous video, I still hope this will make seamless traveling in Google Street View possible. There you even have a new image only a small distance away
Pretty sure that is their intent with this research.
I think this kind of work needs massive optimisation to work in realtime and even more getting without glitches. We all hope they make that but as its now it doesnt make sense how they would go about it, but as with other tech we said there is nothing better yet it still improved in a rather surprising amount of time.
Just two more papers does the line!
Here's a fun thing: if you want to try seamless travel in street view today, try Apple Maps on an Apple device (Apple calls it Look Around, and it's activated via a binoculars button) and then move down a street in full screen. It's buttery smooth and with animations/morphing that feels so much like a 3D environment when travelling down the street - you have to see it for yourself to see how smooth it is compared to Google's implementation.
I bet Apple pulls it off because they record 3D data from the start while Google only has 2D images, and so to compete Google is trying to supplement the lack of 3D data by using AI to generate data/footage for them.
Something that I want to see is an AI that can take video (especially 360 video) as input and create a 3d model/flythrough video of the entire thing that can be played.
I think it's be amazing to have something like a 360 camera running with you on vacation and let you have an insanely detailed record of your trip.
There would more of a point in recording. Not only for memory but to see things you may have missed
This is something I've wanted for so long.
The ability to take a picture, of anything, and just go on forever. Explore an endless, AI-generated scene, and just see what's out there.
One or two papers down the line, I hope to really see this get good, and eventually work in real-time. I want to see a game based on this. No objectives or anything beyond just going around through this AI's mind, seeing 'hidden' parts of an image and exploring strange worlds.
Oh wow, we can really use the Hollywood "ENHANCE!" technique in real life now.
Already holding onto my papers in anticipation
I can imagine Google Earth taking a lot of benefits from it. Imagine if they use drones and airplanes to take photographs of all cities of the world. Then we can fly through and around them with an incredible amount of details.
They are already doing that.
Cause that's not creepy at all...
th-cam.com/video/0QRGrXt0aJs/w-d-xo.html
😓ITS FINALLY HERE😓
Details that are not real and not correct. What is the point? You sound like you treat it like a game. Then why not just playing an actual game? When I use a service like Google Earth, I trust it to show me real, correct data - not stuff made up by an AI.
Microsoft Flight Simulator
It's fascinating to see these AI technologies basically scratching the surface of what is yet to come, constantly improving and advancing.
_"Do not look at where we are, look at where we will be two more papers down the line!"_
Yeah.
It's super jarring whenever I see people not doing that, and expecting models to be more or less the same in the future, whithout realizing how far we've come in the last two papers and extrapolating to the next two.
That coming singularity will probably make people bored of AI generated content. Humans will seek each other out for a genuine life experience... maybe.
@@engelbrecht777 I don't think the singularity will be meant to replace that ever. It only means we can enter a state of "possible" abundance where many things that were thought impossible to solve A.I will assist.
@@carlosamado7606 i never talked about replacing anything. The only thing i know for sure is that nobody knows the future.
@@engelbrecht777 So uh, about that... I've been messing with various types of GPT for a while and I can say that it feels more human than conversing with actual humans...
@Two Minute Papers link to fly into photo please I may have?
The excitement with which you say "Fly" is enchanting.
I can't wait for artists painting a single image of a fantasy city, then have such an AI fly through that city.
Vector Space!!!!! I can't wait, once everything is finalized, every digital media can function as a real-time portal lol!
I can imagine these resolution improvers being added to video players to reduce bandwidth required !! WOW
Like DLSS?
"Hold on to your papers!!" Awesome catchprase, love it. 😆
this is how next gen selfies will look like, I am anticipating this for a while now
Photos are now basically D'ni linking books from Myst
4:10 That is SUCH a trip. First of all, it feels like the equivalent of AI generated music that you see on youtube.
Second, it's feels like a literal lucid dream and it's friggin trippy. Not sure if I'm the only one, but when I'm lucid dreaming and I'm on a familiar place, I start wandering off and imagining new areas as I keep going. That's exactly what this feels like, like the AI is dreaming of new places.
the application of A.i to older films to bring them in HD would be awesome!
your videos are sooo intresting! Keep them up!
Your videos are always so interesting!
I guess the next step is to start with Dalle or Midjourney to generate the photos and then be able to fly around it. And then convert it to a 3d scene. And then 3d print it so you can live in it.
lol
So we can almost zoom and enhance.
This immediately seems like a way to make any painting or image a portal into a new world. Take the painting "Nighthawks" by Hooper and the AI makes a whole city metropolis based on that input.
*_THAT_* is magic.
walking in vr in some auto generated place would be kind of insane
How can I make those 3d images that you are showing at the first part of the video?
Wow. That’s so promising! This will be incredible in just couple years.
Absolutely incredible. Watching A.I. development in this field is nothing less than breathtaking. Just a few years ago the best I had was a 486 DX2 with DOS. Now we get to catch a glimpse of how our own minds dream in the subconscious world, with a desktop grade GPU and a text prompt. "What a time to be alive!", indeed...❤
This is mind-blowing. Love your videos on these things especially because I use AI to create animations, so all of your amazing content helps keep me up to date and gives me new tools to incorporate into my workflow!
This looks like Stable Diffusion and similar AI-generated images. You input a textual description and the system generates likely images. Amazing stuff!
This is so incredibly useful for my master thesis! Thank you for making this video!
very cool. excited to see how immersive photos and google maps, etc will be in the future
Two minutes of silence for the people vaporized on these papers.
love these technical videos
Elképesztő!
Nem gondoltam volna hogy már létezik ez a technológia.
Kimondottan jó lett a videó, a magyarázat mellett, a vizuális részek is professzionálisan lettek kidolgozva.
10/10
For some reason I misheard 'This is too many peppers' hahah. Awesome video, excited to see where this tech evolves!
Thanks for the video!
this could probably be combined with the now-old Game GAN which was trained to represent a small part of GTA in a way that you could actually drive the car around. Entirely within the GAN
Does anybody know where I can find more information about the first tech he was showing? The one where you take a couple of pictures and fly through them
Reminds me of flying in my dreams, I wonder if watching generated content like this could influence your dreams.
this, but more the multi view interpolation stuff, and super resolution stuff, is something I'd like to see applied to computer graphics a neural algorithm that generates geometry from photos, that you then clean/up separate/auto infill, modify and complete in various ways. basically 3d scanning, but better results with less data, and the ability to fill in missing data. additionally, I'd like to see neural nets applied to compress the file size of this geometry, instance repeating shapes/texture types in clever ways, all to crush the filesize as much as possible while maintaining the ability to decompress rapidly, or, ideally, not needing to truly decompress at all! (like with instanced geometry/repeated/layered textures)
always happy when you say hold on to your papers and what a time to be alive, hope you had a great weekend
SEO writer is already a serious job today. Prompt writer and image curators for best AI generation is gonna be serious job in the future too.
I feel like I'm viewing an ultrasound right now. It's not fully complete, but it's beautiful, we love it, we're rooting for it.
Thanks doc ! Thats absurdly awesome
lack of inpainting is a big issue with many old games er (directx8 or custom engines) , that do not natively support the injected-rendering of 2 view frustrums for SideBySide-stereo-rendering for VR, that then just stretch an image instead of inpainting it, to reconstruct a CHEAP second-eye-frustrum (often also relying on a very imprecise depth-map estimate), by using tools such as reshade5.0 (or newer)
So THIS is what Ren has been doing since his show ended. Brilliant.
I remember that in anime Doraemon there's a gadget in the future that can generate manga, we just have to input like main plot, genre etc. I think that it's no longer just a fantasy at this point..
This is cool and all but what are the practical uses for it?
This technology needs to be used to expand old video aspect ratios. For example old tv shows like Star Trek were filmed in 4:3 format, and even though they remastered the original footage for HD, it was still in 4:3 because that's what they filmed it in. Imagine being able to fill in the rest of the background so these old shows could be seen in full HD and in an 16:9 ratio.
Viewing the earth from every angle is going to be 🔥
I am very very interested in games of the future if they take a page out of dwarf fortress and generate a whole world of stories. If dwarf fortress used AI image generation along with this flying into images. You could have a more visually immersive world then dwarf fortress was made with. Especially if the AI was "retargeted" with dwarf fortress' content in mind to keep a cohesive art style. Maybe even include ideas from AI dungeon. We have so much potential in unique virtual worlds. And I can't wait to see how they turn out.
I'm still waiting for the Brain memories turned into images and videos formats.
Really loved it
1:50 It should be noted that the technique shown here depends on image content that is available in other frames in the video sequence.
i knw that 360 degree environment generation is coming but didn't knew this fast. At this rate we will able to generate fully 3D virtual environment juat typing some words. Crazy 🤯
I'm excited to see this technology being used as one more tool in artists' toolbox (rather than replacing their work). For example, being able to quickly remove people from scenes could allow artists to focus on the important stuff!
Cool. Thanks for sharing.
I'd love to test this ai on some of those near photorealistic images out of stable diffusion. An entire world generated from just a prompt.
What about the world we already have?
... and its real with consequences, love, hate, war, famine, fast food and corona. Try it before it's over.
Where is the link to your referenced "previous video"?
Next time i see a spy movie where an agent says "zoom in" or "enhance" i won't roll my eyes anymore.
Another video from my favorite channel, 8 minute papers 😎
Build a time machine because the fly through videos are the best screen savers
Any time i view this videos i find AMAZING PROGRESS ... i view all this like a child looking at a window of a chocolate shop.
Glad to know Igor got out from under that so called doctor and finished his computer programming degree.
The randomly generated universe of Star Citizen becomes more possible each day.
Bro where do you find these papers?
Do you have a tutorial on how to set this up and use it?
This is becoming a brand new way of creating video game works without defining vertices and shapes all individually
This is some awesome stuff
I will use it to find my dad.
Seriously though as a person who worked on a neural network model and creating a clean dataset, when you mentioned how the dataset for this paper is my brain froze for a bit. A really amazing hard work from those people.
Also thanks a lot for keeping us updated with the new AI tech.
I can’t wait until this sort of technology can be used to replace video compression. Imagine the creator making a very low quality render of their movie, meant to be uncompressed with a particular AI algorithm which can upscale it 50x with perfect accuracy on the user’s end.
That’s a very interesting concept. At some point maybe you’ll be able to stream Netflix in 5,000k 4D via a rural connection. I wonder if every movie would “recreate” the same for each viewer or if there would be small or not so small differences.
In that scenario, YT will still have compression artifacts of course LOL.
Let’s use this to build historical recreations of famous places, buildings, events, to be seen inside vr goggles. Imagine a drone flight through the city of Knossos, seeing buildings, people, marketplaces, ships, weather, etc. imagine being present for Krakatoa exploding. Imagine seeing the moon landings from any angle and with accurate backgrounds. Imagine being with a ship full of Vikings landing in North America. Walking in the rain on a Paris night in 1902. Or a completely abstract environment inside the metaverse.
Your diction, the way you speak, reminds me of Formula 1 journalist Walter Koster when he questions drivers at press conferences.
Where can we use the multiple pictures to fly video or is it not public?
I thought about the amount of images that were shared on social media of certain landmark locations (from different angels) and if AI could join them altogether to create a fly-through.
i can see that being most useful for street view. making the virtual camera move smoothly as if they provided a video instead of photos every 10 meters
I'm hoping this becomes public someday. This is so awesome!
just two weeks ago I thought about flying through Google Streetview in a smooth way and how close we might be.... now this
i seriously hate how you talk and inflect certain words, but your content is too good to just stop watching
The way you end every word with that singing thing 😅
Fascinating! I think that I’d use this technology for visual effects in both film and video games. Adobe is already using AI in its software so it should be exciting to see where they go with it down the line. The pace is amazing.
I'm curious how well this will work with fictional images for input. For example, what does the back of the Walt Disney castle look like and what would happen with the Marvel Studios logo?
Can't wait for the next paper on this Karoly!.
The thing I would like to see is environment generation with this technique.
You could create your base scene to get a theme and idea going. You could then eventually use these techniques to generate an environment to support your starting idea.
Further advancement with this type of technology could include fully generated environments from a group of objects.
Let's say you have a set of 3D models (maybe a type of tree you like, a type of grass or bushes, maybe some rocks or cliffs) and have ot generate a custom environment based off the models passed in.
There can be a lot of cool ideas that goes beyond just expanding around or into an image. I think this type of technology will be the stepping stone to a whole new world of ideas.
I love this!