This AI Creates Real Scenes From Your Photos! 📷
ฝัง
- เผยแพร่เมื่อ 21 ก.ย. 2020
- ❤️ Check out Weights & Biases and sign up for a free demo here: www.wandb.com/papers
❤️ Their mentioned post is available here: app.wandb.ai/sweep/nerf/repor...
📝 The paper "NeRF in the Wild - Neural Radiance Fields for Unconstrained Photo Collections" is available here:
nerf-w.github.io/
Photos by Flickr users dbowie78, vasnic64, punch, paradasos, itia4u, jblesa, joshheumann, ojotes, chyauchentravelworl, burkeandhare, photogreuhphie / CC BY
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Haro, Alex Paden, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bruno Mikuš, Bryan Learn, Christian Ahlin, Daniel Hasegan, Eric Haddad, Eric Martel, Gordon Child, Javier Bustamante, Joshua Goller, Lorin Atzberger, Lukas Biewald, Michael Albrecht, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Ramsey Elbasheer, Robin Graham, Steef, Sunil Kim, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil, Tybie Fitzhugh.
If you wish to support the series, click here: / twominutepapers
Károly Zsolnai-Fehér's links:
Instagram: / twominutepapers
Twitter: / twominutepapers
Web: cg.tuwien.ac.at/~zsolnai/ - วิทยาศาสตร์และเทคโนโลยี
I'm sorry sir, this was too good and I could no longer hold on to my papers.
a crime punishable by death. A shame.
Papers are held on to until you throw those babies out like doves at a wedding.
This is so sad
me too... i dropped my papers all over the floor and couldnt stop
I dropped so many of them as well..
running this AI on all google street view photos for a seamless street view experience
That's what I was thinking. They already use aerial photography (expensive) to build beautiful city models. Imagine what they could do with all the pictures they have access to.
Oh god, that would be amazing!
@@mnomadvfx Dont they just reconstruct it using satellite images? (from oblique angles)
Leave that, think of the possibilities for animation softwares like blender, it's gonna be a game changer
how gta 6 be like
They look like they were carefully scanned, so cool!
As a filmmaker I'm looking forward for these photorealistic assets to use in film projects. :D
Seeing these papers evolve and truly understanding their background has been amazing. 2 years ago I had no idea what was happening but just liked the videos cause they were research and sounded smart, now I even know the previous problems that this papers are overcoming and I can feel it like damn
You are very kind, thank you so much for the heartwarming message! 🙏
Think about who is going to plan an entire professional career on 3d modeling, lighting, texturing, materials, animation, vfx, etc or, worse, who has just started the profession having spent thousands of hours practicing, for this people thinking 15-20 years in the future could be a nightmare. I imagine that at some point this AI will surely become aware of the general laws of physics, not in the sense that they can explain and write down a physical representation of reality by theories and equations, but in the sense that they can comprehend - just like a human being can easily do - how a white plastic sphere inside a cubic room with red and green walls and a 100-watt lamp in the ceiling should looks like, only by a text description, or a sketch, or a single photo, and recreate a realistic scene on the fly. At that point (we could also extend the concept to various other fields and industrial processes) we'll be in serious trouble, assuming that the economic and social capitalistic production relationships remain the same as today.
@@ittixen haha you're fantastic, thankyou... I'm waiting to see these experimental works evolved in real professional applications for daily production... I paste here a previuos comment of mine, thankyou ItamarB.
------
I am not so optimistic about the future of the traditional work of the modeller. And i don't think about the reproduction of models of already existing objects to be scanned, but about the invention of completely new models with unreal subjects. I believe that the revolution will be inevitable and will come from artificial intelligence. These systems will not think about how to create 3d faces, chamfers, texture projections, etc., but they will directly know what is the nature of the object to be reproduced, they will know what is a dragon, what is a spaceship, etc., learning and comparing from millions of objects already created by humans and obviously they will be able to create new ones never seen before. I think that in the end they will come to a procedure based on voice or text commands, in which all the work will be performed by a technical director who will say: create me a night scene, similar to this one where Terminator arrives in the 20th century, but under the snow, the outside temperature must be very cold, I want terminator thinner, I want him of Chinese ethnicity, like this actor but younger, with beard, put him in red underwear, make him walk 5 meters, limping a bit as in this scene of this western movie, now he must be assaulted by a python 15 meters long with the skin of this chameleon, let's bring the whole scene in the middle of a polar ice shelf as in this photograph, etc. etc.. It is possible that the artificial intelligence will build a 3d space to set the whole process, but I don't think it will be a geometric system like the present one, it will be developed by the computer as we visualize it inside our brain. This will also be used for engineering design, for example an architect will provide the requirements of space, cost functionality etc, he will be able to sketch by hand a prospectus, or start from a pre-existing design, or manipulate virtual masses in space, and the artificial intelligence will develop the project autonomously, following the inputs of the operator who will correct everything along the way according to his vision. Obviously the automatic process will touch the entire value chain of production, including the disappearance of bricklayer workers, perhaps it will also touch the creative and ideative realm. In a communist system of production, based on the absence of money and centered only on the use value of the product, this is a paradise, but in a capitalist system based on the exploitation of the human wage work, it is a kind of nightmare. In an imperialist system then, like the present one, this foreshadows the darkest geopolitical scenarios.
@@zdenekburian1366 Spot on!
Can anyone recommend a channel like this one but in cybercecurity ? Would love to gain this kind of understanding
Two Minute Papaers : What a time to be alive!
Me as a 3D modeler : Dead, dead deaaaad, dead!
automatization will make mundane jobs soo easy that it will considered as hobbies, good thing there will be UBI
@David Amaya survive, adapt, and overcome the AI
Well fictional elements will still need modeling
@@ale-pg3xb Until AI knows better than us what fictional means! :D
Saidou Bâ well ai needs training, and the interesting thing about this is that it creates models from a very restricted range of input. But yes, maybe sometime it will.
I like many channels on TH-cam but yours is the only one I am subscribed to because not only do you talk about some really amazing stuff, but also keep the length of your videos short ( and thus , don't consume too much of my time ) . Keep up the great work.
You are very kind. Thank you so much!
@@TwoMinutePapers Your yotuube channel is the greatest one Im subscribed to.
omg imagine shooting fight scenes in a movie from a few angles, running each frame through this method, and then being able to choose ANY camera perspective with a vr headset.
maybe two more papers down the line :)
Oh damn 3d movies from 2D. I could actually see this happening.
10 Papers down the line, you write a movie script and the ai outputs a full movie, along with a tool to change some angles. What a time to be alive!
already available:
www.intel.com.au/content/www/au/en/sports/immersive-experiences.html
th-cam.com/video/xYudMimnmAg/w-d-xo.html
th-cam.com/video/J7xIBoPr83A/w-d-xo.html
th-cam.com/video/KEr62kr5Cys/w-d-xo.html
th-cam.com/video/HTUu_JDP-0s/w-d-xo.html
th-cam.com/video/NhVrfFlqtn4/w-d-xo.html
@randomguy8196 Well let's say you have multiple cameras recording the same shot from multiple angles, moving objects would be stationary as they are at the exact same place on every camera for a defined frame.
can't wait to see the entire world virtualized in 3D on Google Earth thanks to street view and this AI
But... It already has 3D
@@circuit10 Only big cities
@@kryplus5471 True, and it's low-res, but they don't have to be that big, my house is in 3D and I think I'm in what would be called the suburbs (I'm not sure though)
@@circuit10 I'm hyped because I think that Google has enough data to realize this, with tons of satellitary images, street view photos etc, one day we could explore almost all the places of earth in 3D, and turn this even in videogames like Microsoft Flight Simulator 2020 recently did but with worse results.
You might wanna check the new MS Flight Simulator :)
crime scene: happens
this AI: hold on lets do a CSI-3D reconstruction
and ZOOM IN !!!
What a time to be alive
🦲🚶🗑️✔️🔥
🙌🙌
@Jordan Kane booo go back using sticks and sharp rocks 😌👎
🙏
Truly.
Imagine being able to see and explore the past this way, like taking a photo, illustration, or painting and putting it in VR and exploring around stuff. That would be sick.
The last time I was this early, I held onto my papyrus
and it hated puns
this joke is very punny
I was thinking about the paper itself, papyrus. But it seems it's no longer the case...
@@GregorianMG *takes off phone case*
2020: We are able to recreate 3D scenes from a bunch of 2D photos
2050: We are able to recreate Fully interactive holograms from a video
2100: I made a 1:1 virtual world simulation from a pixel
More like 2021 for the second one.
2075: I enhanced the reflection in his eye to determine the killer
Plenoptic versions of this are already being worked on so 2050 seems rather pessimistic.
"Enhance", "Enhance"....
We have been able to do that for a long time. Though not through this method.
Your videos blows my mind every single time. The development go so quickly, and i was already really impressed by the last version!
It would be cool to see this done with videos too.
For example if we could take all the peoples videos of the Beirut explosion and it would create a 3D version for us to watch.
Maybe 2 papers down the line?
This is what I proposed for my entrepreneurship course, but the technology was financially unviable.
So it is indeed up to science to do the work!
The AI recreates the interior too and we can know who is responsible for the explosion?
@@manda3dprojects966 How do you suppose that would work lol?
@@manda3dprojects966 it's not a who but a what, it was a large amount of badly stocked fertilizer that exploded, we dont realy know why it exploded but it wasnt on purpose
If you could process that data quickly, police could use it by taking livestreams from events (like protests) to more intuitively track a suspect fleeing the scene (I'm thinking of that antifa shit who shot a man dead in Portland). I'm guessing you would need a hell of a lot of video of the same areas though.
Of course, you could also just use it to even more easily track people in general, if you want a human to be able to interpret the data easily. Yay. My bet is on China implementing it first. They certainly have enough cameras and just surveillance in general everywhere.
Any upload by you is a joy to watch.
It has taken me a year of watching your videos, but I feel like I'm finally starting to understand how quickly this field is growing and the implications of the research papers you share. For the first time, I felt like I really understood why you exclaim "What a time to be alive!" at the end of every video.
wow! I need this so bad!!!!!!!!!
Same
This is interesting. Hope the Paper will give enough info on how it's done 😀
No you don't. You want it but you don't need it.
@Mateus Luan Maciel bruh xD
@@Mateus01234 Truth.
I can't wait for commercially available photo-scanning software making use of these techniques, will change a lot for sure.
Everybody's gangsta until you turn the camera around, and realize it generated the entire planet.
This is so advanced! I can't wait to apply this tech to stuff, so many possibilities!
This channel is fantastic. Thank you for making these videos.
this is insane, i feel like im really close to the future the kid me thought of, amazing.
Sometimes basic things like a pen and paper can be more creative and powerful than a future AI that does everything for you.
Imagine having 3d Photography that you just took.
Just changing the Angle alone for what you want it to be is going to be so useful.
Those lighting shifts are breathtaking
I'd love to see a series or separate channel where you go over the latest AIs in different fields, show how to set them up, how to duplicate the results and then how to use them on your own projects. Right now I'm trying to figure out how to apply an AI to a certain project but it's been a bear.
I cant. I'm so amazed, the AI technology is advancing so fast. I'm so excited for the future. My papers are all over the floor.
I want to see a virtual world in VR all made up from 3D models generated from just a couple of photographs and filled in with details of a tuned artistic AI algorithm.
so cool. this is going to make photoscanning so much easier!
I really want them to showcase more plants and similar very thin, translucent items.
People removal is nice, but this is what would REALLY get me hyped.
i want this so badly every video of your videos man just make me EXTREAM HAPPY from what am seeing computers can gen i hope this could turn in to full data app's
this is out of this world man, incredible
Gotta love the exponential pace of innovation. :-)
What a time to be alive!
Stunning results!
a moment of silence for our fallen brother MS Photosynth
I'm excited for people to develop this for Photogrammetry purpose
Honestly this AI seems like such a forward step in the movement of vr we're going to take it for granted in the years to come!
Amazing, can't wait to play with it!
This reminds results of photogrammetry/sfm. I wonder how the both could work together. Maybe this method would aid achieve better results with heavy lighting variations of the inputs.
Mind blowing man!
This is better than photogrammetry. A use case for this would be in VR. Imagine using your old photos you can move around it in VR.
Me, re-watching Minority Report: "Ah, it was a simpler time..."
I love doing photogrammetry scans, I NEED THIS
Research is great but we need products .. this would be such a game changer in virtual tourism (VR)
Good that you included the disclaimer :)
Truly sci-fi papers, two more papers down the line and we might get this on objects that change poses on each photo like celebrities!
😍😍😍 neural light field rendering and 3D reconstruction
This will revolutionize photogrammetery
I already was impressed when we started with desktop 3D years ago, I felt like it (3D) should be implemented into every level of learning and teaching and that we were truly living the second renaissance but now a days with machine learning, my mind is just blown out of the park.
photogrammetry could transition from something only the pro could do only in perfect lighting condition
into something a thing you would do to apply Snapchat filter, this is amazing
Wow... at some point we'll just feed an AI a couple of old photos and it'll recreate the house we were living in as kids in virtual reality. And then if at some angle that wasn't in the photo things don't match with our memory, we'll just tell it "no, there was a stairwell there" or something and the AI will create it.
Imagine where we are in 30 years!
Can't wait to see what this might do to a great painting, if the projection can create mobile depth of field and movement. WOW!!!
researchers from 6 months ago: "its N.e.R.F or nothing"
this paper's researchers: "hold my papers"
hahahahah,
now its nerfW or nothing
there is another model that I really like and its called Neural Sparse Voxel Fields and its look amazing and much faster! I want to implement it but after I finish Nerf-W implementation. I think they can be combined together to one model, and I have my own idea to improve them. hopefully it will make it faster and more accurate
@@eh_channel3 cool send the link to the research paper or whatever, I wanna see the result.
@@person-s4743 the paper for Neural Sparse Voxel Fields
is here arxiv.org/pdf/2007.11571.pdf
My idea to make it better is to use something else to represent the scene. I dont want to get a single voxel (and color) with the model, I want to get several voxels per run or represent the scene little bit different. I plan to use curves for that, but I'll work on that after I'll finish with NSVF (hopefully I'll manage to combine the changes from NERF-W in NSVF without any problems)
@@eh_channel3 i dont understand what you said, even though I am learning ML, but okay im reading the paper
this needs to be implemented on streetview now!!!! amazing
Dr. Feher, What a time to be alive !! Ok, What happen if one put more detailed photos ? I mean, take the most of photos possible and in good resolution.
Imagine just taking a few pictures of something and in a few minutes getting a high quality .obj file of it on your pc
What a time to be alive :)
Good god!! Software is progressing way faster than hardware can catch up!!
I imagine this technology would augment or even supersede modern photogrammetry scanning methods. I'm so excited for this stuff to help create expansive worlds in video games.
some day objects and people from photos come alive for a short time - it would be beautiful!
I am pretty sure I saw a similar method 2 years ago, specifically doing it with some giant statues in South East Asia as an example. Like photogrammetry and not just synthesis of new angles.
I was originally looking for methods that estimate geo location from single images and short videos based on Google maps, the sun and some cultural guesses. But for lost in the references.
I can't wait to see this being implemented in real world applications such as VR and map views!
Truly, what a time to be alive
Thank's for your job! Could you tell how many photos this model took to make?
The paper lists the following as the training image numbers:
Brandenburg Gate: 763 images
Sacre Coeur: 830 images
Trevi Fountain: 1689 images
They're from a larger phototourism dataset, so no specific images were taken for this, just the unusable ones filtered out
It would be amazing if we could use this in Meshroom!
holy shit, I feel goosebumps. Imagine where this tech will be in 5 to 10 years.
What a time to be alive indeed!
Outstanding!
Astounding!
Mindblowing
Imagine future phone camera AI and hardware that will make all of your photos explorable in 3D.
Any application for 3D scanning smaller objects?
That sounds like something this could absolutely handle.
use photogrammetry for now. I recommend Agisoft metashape!
AliceVision Meshroom works for me.
Agisoft photoscan
I was just thinking about this for month, google will like it for sure.
Can not wait to see which photogrammetry software adopts this into its system first.
I'm ready to count the "hold on you to your papers" again in this video
This is amazing, if you could export the black and white image at 0:55 , one could make stereograms of the image to mess with people
so can someone help me, is each scene in this video from the training set and it just learns to produce a representation from all training set images for that scene? or does each scene represent a multi-dimensional representation created from a single, out-of-training-set input-image ?
They use differential volume rendering to create the scene, the training set is to teach the AI how to cast light. So my guess is it's the latter, but I could be wrong.
0:22
I'm confused by that, too. He says that training can take hours or days but needs only be done once and then rendering is quick. Does that mean it takes hours or days for a given scene / set of photos? If so, this is really good (but slow) photogammetry. If instead he means that you train the neural net once and it can 3D-ize any scene, that's more revolutionary.
Someone needs to make this into a consumer program, like right now!
I would love to see an open source software package that included many of these creative AI algorithms.
It would be like an entirely other level of photoshop or blender
As you say: "two papers down the line"
This paper is just the first and so much has happened. Most probably the second in will remove the very few the problems this has and we could finally see this technology in its full glory. What a time to be alive!
Google street view with this technology + simulating people and cars = GTA6 :D
I could see this tool being used to make 3D models of locations, or to create hyper-realistic scenes that 3D models could interact with which could offer marketing a better way to show off a product and not have to worry about have a massive stage/studio to work in.
wow, incredible!
This is the most important channel on earth for year 2020+
This is going to be great for VR / Video Game asset creation. Especially if tons of virtual objects are uploaded to the public domain.
This is a game changing technique
Google Photos, Streetview, Aerial Photography and Satellites combined are probably a 95% accurate model of the world with an AI like that.
WHAT A TIME TO BE ALIVE
I wonder if it'll be possible to do photogrammetry with this method and get a 3D scene you can use in blender or similar
Thank you!!
Crime scenes about to be LIVE.
Amazing Great work. I was thinking, maybe some WW2 footage may have enough "samples" for reconstructing past buildings etc? I dunno. Just a thought.
Imagine this used in VR applications like Google Maps VR.
First lightfield reconstruction paper (nerf) is based on Fyusion's LLFF tech!
woah technology !!
Amazing!
Two more papers down the line, this might run in my smartphones's gallery in real-time! :)
I wonder how well it does with black and white images. Imagine recreating old photos!
Take that, Meshlab!
This is extremely excellent! Perhaps with Covid-19 there will be -- or are! -- virtual travel or tour sites for us to visit from our lockdown locations!!
I bet in the future the occlusions such as the dark recesses of the arches will be improved upon by increasing the brightness -- the way I can manually do -- and deriving the 3D content, then knowing that include it - but backing off the brightness. I encountered that just yesterday when looking at my one photos - I had one of what was obviously a giant tesla coil spark, but you couldn't see anything else in the photo. When I cranked up the brightness, I could see individual people, and other details totally obscured in the original.
what a time to be alive