Ken Burns Effect, Now In 3D!
ฝัง
- เผยแพร่เมื่อ 7 พ.ย. 2019
- ❤️ Pick up cool perks on our Patreon page: / twominutepapers
📝 The paper "3D Ken Burns Effect from a Single Image" is available here:
arxiv.org/abs/1909.05483
The paper with the Microplanet scene at the start is available here:
users.cg.tuwien.ac.at/zsolnai...
Scene geometry: Marekv
Image credits: Ian D. Keating, Kirk Lougheed (Link: www.flickr.com/photos/kirklou... ), Leif Skandsen, Oliver Wang, Ben Abel, Aurel Manea, Jocelyn Erskine-Kellie, Jaisri Lingappa, and Intiaz Rahim.
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Alex Haro, Anastasia Marchenkova, Andrew Melnychuk, Angelos Evripiotis, Anthony Vdovitchenko, Brian Gilman, Bryan Learn, Christian Ahlin, Claudio Fernandes, Daniel Hasegan, Dennis Abts, Eric Haddad, Eric Martel, Evan Breznyik, Geronimo Moralez, James Watt, Javier Bustamante, John De Witt, Kaiesh Vohra, Kasia Hayden, Kjartan Olason, Levente Szabo, Lorin Atzberger, Lukas Biewald, Marcin Dukaczewski, Marten Rauschenberg, Matthias Jost, Maurits van Mastrigt, Michael Albrecht, Michael Jensen, Nader Shakerin, Owen Campbell-Moore, Owen Skarpness, Raul Araújo da Silva, Rob Rowe, Robin Graham, Ryan Monsurate, Shawn Azman, Steef, Steve Messina, Sunil Kim, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil.
/ twominutepapers
Splash screen/thumbnail design: Felícia Fehér - felicia.hu
Károly Zsolnai-Fehér's links:
Instagram: / twominutepapers
Twitter: / karoly_zsolnai
Web: cg.tuwien.ac.at/~zsolnai/ - วิทยาศาสตร์และเทคโนโลยี
These videos are so underrated. Thanks for making this information available for anyone to understand!! And your enthusiasm for the subject is contagious!
0:52 HOLY MOLY. Sleeper paper! A possible application of this is for VFX, 2D "conversion" to 2.5D or parallax 3D is done extensively in VFX, for matte work(background plates) and can be very tedious work similar to rotoscoping to get everything masked and cutout properly then align everything then animate it all in sync.
If this technology is used in Google Map's street view, it would be awesome!
Sudipto Chandra Apple’s street view uses it
You mean google earth studio?
@@mattijs4222 Yeah. My mind was blown when I first saw the effect in Apple's Look Around. It makes nearby objects like cars, street signs, and pedestrians wobble slightly when you pan the view around.
Lumumba B. Was wondering who developed that parallax effect. What’s the source that says it was Adobe?
can't you just feed it a picture from Google Map's street view and do it? It's not as convenient but it supposedly works on any image so...
The phrase "Dear fellow scholars" has very quickly conditioned me to become relaxed and ready to be educated.
i grab my phone a little more firmly
Yo Karoly, What you're doing is so useful, beautiful, and impactful that you should have more youtube channels dedicated to other fields of science. You can make a media company and hire professionals from different disciplines to review scientific papers. Right now, there are no youtube channels that review engineering, physics, medical, chemistry, biomedical, pharmacological, etc... journal papers.
Karoly - this idea would be awesome. Call it the "Karoly Paper review channels". Make sure you do not deviate from your existing format.
i thought since you liked him so much, you would be one of his patreons, rip tho
The main problem is: it's hard to find such people as Károly ;) Basically, it would be at least a half-time job (at the beginning probably more like a full-time job, because it takes time to learn how to edit videos, etc.). As for chemistry and physics: you would probably need several people specialized in different fields of chemistry/physics. Machine learning is growing, but compared to natural sciences, it is still a rather narrow field.
There're a ton of exciting papers that are not accessible to people without deep background knowledge in the subject.
Also, not everyone can be a good narrator. So, there're a lot of small things to fuck up. Besides, most Ph.D. students just don't have enough time to do something like this, they are already overwhelmed by teaching assistantships and conferences. All that makes me appreciate Károly's effort even more. All in all, it would be cool but it's just a very hard job that is also not very well paid.
Depending on the subject area though you don't really have nice visuals to use and the subject matter can be very esoteric so it would be very difficult to make videos about.
@Connor Jones all papers should have atleast something!
This is a misuse of the term "zoom". When perspective changes (camera moves along the longitudinal axis) it's called dolly, not zoom. Zoom can perfectly be done in 2D, as it only involves a change in the focal length (vision angle)
Well, Technically not. After years of fighting for the need of professional vocabulary in normal people on youtube, I've finally understood this apparently :). people learn about the end result and associate it with the term. Zoom is what we attach to the object getting closer and NOT bigger. So You might wanna take that back :)
@@riquardi Are you suggesting that a "zoom" lens moves the camera closer to the object? :)
I think your point is prescriptive grammar vs. descriptive; "if a lot of people of people are using the 'wrong' term, that has become the correct definition". I do not agree with that opinion unless a *supermajority* of people are using a single term unamiguously. Though a huge portion of the English-speaking population use "literally" in a hyperbolic fashion as a figure of speech, that does not mean that "literally" now means "figuratively" as its primary definition.
If a bunch of illiterate children use the term "zoom" incorrectly, does that mean that the definition of it becomes their usage? No. Zooming in still means visually enlarging, not getting closer to.
@@gavinkistnerus3895 read your points, as you said in third para, yes Incorrect usage by people won't change the definition of terms at all. What I'm trying to convey is the placement of term 'zoom' in the language is CLOSER for the 'effect of looking at the object as of higher definition/ closer/ bigger than the respective terms like Dolly. WE know what these terms and usages are, but a random person refers to the act of a small object getting more 'clearly visible' (High definition), bigger on screen, which then also means pulling it closer...as "ZOOM". This is independent of any other topic or issue. Hope we are clear on this now :)
@@gavinkistnerus3895 by the way, are you the Gavin who lives in Colorado? Boulder? Just asking
AI: "I'm about to end this man's whole deer"
This is awesome. I can't wait to buy some software product that converts my stills into dynamic motion for slideshows. This is revolutionary.
What a time to be alive ! And we have seen nothing yet ♥♫
Now my powerpoints will have depth maps of cats!
Further give those great looking 2D bar charts depth so you don't have to use uglier 3D styles
Abraham Mast This might be of interest to you: th-cam.com/video/_3loq22TxSc/w-d-xo.html
What a time to be alive!!! :))
Next feat: creating a city map from a reflection in a coffee cup.
LMFAO ... exactly.
I always envisioned that in the future, technology will be able to turn 2d pictures into 3d models that can also be experienced as stereoscopic VR, AR scenes.
This is one of the first steps to achieve that.
So guys, take photos, a lot of photos so that if you or your family want revisit your past in the future the raw material would be there.
@@beedykh2235 Now that is some "forward thinking". Very nice.
@@beedykh2235 photogrammetry does this already plenty of papers cover on futher research into it on this channel as well.
Oh god, I can imagine an algorithm filling in the back of a model extracted from a front facing photo with another face. That would be unbelievably creepy.
@@WLxMusic you can already see something similar using photoshop's content-aware feature by deleting features on a face and the algorithm filling in the blanks. Trying it on a large crowd photo might be exactly the horror show your imagining.
"What a time to be alive!"
I love this guy.
Your papers are always so insightful!
Technically modern phones have at least two cameras, therefore depth estimation is possible without a special setup, contrary to what you've said. Nevertheless, this paper is indeed stunning as its approach has no restrictions for the camera system
It's also more general. Phone cameras are really close to each other, depth can only estimated for close objects and without much resolution.
Also you'd ideally use two identical cameras, otherwise you have to start correcting for stuff like different aperture, different focal length, diffetent field of view, etc, which would definitely make things more messy
you still need that algorithm for filling in missing data with such a dramatic zoom level. Phone cameras are too close together to cover each other's blind spot
My phone has a ToF camera as well as 3 other cameras on the back, do I get better depth mapping?
@@SuiYo Yes, that's how the shallow depth of field synthesis works, and is the reason they have it in the first place. Also, look at kinetic game controllers -- they produce depth maps and that is what the API gives you.
i love your general enthusiasm and excitement for this stuff. thanks for the great content
Pretty spectacular work. Thanks for the video
Been watching your videos a lot for a month now and just realized I wasn't even subscribed yet! Well, you got one more subs now. Keep up the great work!
i really hope they release this type of stuff for rotoscoping ASAP
This will be extremely useful for news videos and other stuff where they show images to illustrate a topic
Hey, just wanted to let you know that I appreciate the longer and more detailed description in this video. I wouldn't mind future videos being as detailed. Good work!
amazing summary as always!
We use camera or projection mapping in 3D programs to do this without AI.
Adobe's After Effects has some funky tools to assist even more.
Vanishing points in photoshop where also around since ages.
But its great to see we get this for free now. Same as we don't have to ride freaggin' horses anymore...
mindblowned
Being able to see your videos makes me think -> What a time to be alive!! :D
Cool, I just finished watching Gaëtan Boutet channel on how to animate 2D images. This AI tool will make things easier when it becomes available to the public. Thanks for the video
What a time to be alive!
thank you for this! i see so many photos of this on twitter but had no idea how it worked until now :)
do u know what is the name of this software
Cool! GREAT vid!
Oh my god, this is one of the coolest papers this year
we have always made manual (3D or 2D) depth masks and background cloning to achieve the effect.
Can't wait to see this on Street View
YES! sew it all up
If you have an iPhone or iPad, you should try the "Look Around" street view feature in Apple Maps. It actually uses this sort of technology already. It's very subtle but natural. You'll notice that things like cars, street signs, and other nearby objects will seem to wobble as you pan around. It's pretty new, so it only works on a few big cities like New York and San Francisco for now.
I see this used all the time but nothing as good as what you’re showing here; amazing stuff as always, you have an awesome job.
*boom* that was the sound of my brain exploding in awe...
I love this channel 💗💗
Awesome!!
You deserve 100 million subscribers. That's how awesome your content is.
What a time to be alive! :)
@4:02 "Lllllllllove it."
This + VR would be huge
Your videos are the best! Such cool stuff AI is bringing to the table! Can't wait to see what's next!
You're doing an amazing work by breaking down such cool research papers in just 2-minute video for us. ALLAH bless you.
Welcome to 2020 where our phones get depth estimation really spot on. But some also use a IR projection parallax to reconstruct the depth map.
You could avoid the in painting by some geometry adjustments on the depth axis and the camera perspective
Wow! Great!
What a time to be alive indeed...
I love this channel
What a time to be alive boys.
Now you can view animated images in PDF files?? MIND BLOWN!!
I love the Ken burns effect so much that I can't upgrade beyond Mac OS snow leopard
Wow if we can make observations without measurement we will finally be able to see Schrödinger's cat. I cannot wait to be able to pet it once this technology evolves a couple of papers down the line.
Love it
Such a cool TH-cam channel !
Nice!
I think it would be better if they used the already segmented layer and go through an image restoration to restore the black space would be perfect .
What time indeed!
What a time to be alive... :)
So amazing. Cant help but remember Elon Musk speaking to only needing cameras for self driving systems and lydar was unnecessary.
He is correct in that a lot can be gleamed from 2D images if built carefully
The inside-out tracking of Oculus Quest also comes to mind.
Cameras, ultra sonic and radar are all currently used on Teslas. Cameras don't overlap and thus can be compared to driving with one open, though. They're used to recognize objects and to judge distance, though with small accuracy.
@@nagualdesign yes, your right, it's using 3 overlapping cameras. if you can with one image, it's better with two or 3 and even more if you have ultrasonic and radar. Also the side cameras dont overlap but they can see depth too, the ultrasonic sensors can help here but not at high distance like you can see with this video: ?v=y9Kcn4fCDJ8
but of course you don't have to be as accurate as a lidar, you just want it to be as reliable for the task of FSD : seeing things and their distance, not representing a highly detailed 3D model of an object.
@@nagualdesign I think you'll find the eye focus muscles can't distinguish anything past 50 feet, and barely different at 20 feet. In strong daylight (smaller aperture) the hypofocal distance of the eye is quite close.
"Zoom and enhance!" is now a thing.
Remember when we all laughed on how that thing was a purely fake thing in sci-fi movies and shows?
Does it enhance though?
All it needs is a few manual touch ups for flagpoles and stuff, would be short work in photoshop if they can export their auto generated depth map as a greyscale image then import the changes.
The moving wall paintings and newspaper images in Harry Potter movies come to mind.
plugins for this have existed for years in video production softwares, though this might improve on their quality
Yeah, this paper was essentially outlining a different technique to get better results in areas where previous methods were a bit inaccurate
How well do these previous methods work? Do they need some manual work (e.g. drawing outlines around a object)?
@@karlkastor the entire paper is in the description of the video, if you need details
Karl Kastor havent used them personally but yes i would imagine having to manually identify and roughly outline objects with some form of smart selection tool would be involved. not an awful lot of work but also not very easy either.
something like this
th-cam.com/video/CHuOT_KVaLE/w-d-xo.html
Newer phones *do* have "depth" sensors. It records Time-of-flight of an infrared flash in low resolution. I was just playing with my Samsung Galaxy: the built-in augmented reality measurement app is amazing. Third-party apps include automatically making floor plans as you pan the camera around, or make a full 3D model as you walk around a complex space.
The built-in camera apps store RGB-D in a proprietary format, using with the paired viewing app.
This could definitely be used in neuroimaging. So cool!
I wonder if this could be applied to billboards in 3D games. Like if you had a mountain covered with trees, you could take a picture from a location where your player is near and just render the one texture with this effect rather than hundreds of trees.
This would be so cool for slideshows
Combining this with some tools to hand fix problem areas would be amazing. I imagine AI will revolutionize the workflow of film in in 5-10 years.
For animation I imagine being able to move the camera around more freely, add lighting and shadows, draw between frames, lip sinc etc.
There are more phones coming out with TOF depth cameras recently. I have one and I love what can be done with it, it opens up a whole new world of possibilities. You can't beat a TOF sensor for accuracy and speed. I get full 30 fps millimetre accurate depth data on my phone for AR, 3D mapping, etc.
However the depth sensors are not perfect. Limited range, depth shadows, holes especially with dark or reflective surfaces, etc. So there is still a place for something like this. For close range though I think TOF is better.
I got a Ken Burns Masterclass ad before this.
Make this into a plugin for FCPX, Resolve (Fusion) and Premiere/AfterFX and sell it. Loads of videographers will happily pay to not have to manually create this effect for every still in a video.
I'm curious, what a room with a big hyperrealistic painting would look like if one would apply this software. I suppose the result might turn out pretty trippy.
is the source code or a demo available?
Not yet, the author wants to according to their comment on hackernews but needs approval since the work research was done at Adobe.
Authors website sniklaus com /papers /kenburns
the website also has link to the hackernews submission with some very interesting discussion of how much traffic is generated when papers like this are published (500 gigabytes of traffic in a few hours).
There's also link to s shader toy doing something similar with hardcoded geometry.
As a $1 patreon supporter (still counts, right?) I think you should do a video about Poet which I feel is the birth of a whole new field in AI
This are the guys that beat the Montsezuma challenge, a game no one could beat.
For the rest, I really recommend seeing the presentation Jeff Clune gives
Presentation: th-cam.com/video/CsizpWJ-nD0/w-d-xo.html
Paper: github.com/uber-research/poet
I'm most interested in the depth from color algorithm. this has a lot of potential applications
As cool as it is, it just makes the whole scene look as if it's made of cardboard cutouts. Especially the first example provided: It makes it look like the background foliage is a texture on a wall.
It's like what they do with old pictures in that Oak Island show.
My guy, you said your name and the chair next to me started moving!
for landscapes it could eventually project a virtual map of the scene depth and then poll Arial photography catalog to fill in missing textures from associated matching GPS positions. Sort of like taking the textures from google earth to fill in areas behind identifiable landmarks. we are living in the future!!
In future we might be able to walk around a VR envinroment generated by AI just from one photo.
How amazing it would be... We could presence the past.
How cool is that 💗
@dustboxednorth Space car is from a different film though.
As for "flying car", closest you can get as of yet would probably be an autogyro. Though keep in mind that even though its supposed to be a safer design than other hobby aircraft there are more accidents with them than other designs counted against flight hours. Whether that's from poor quality, poor education, overestimated design or overconfident pilots... Well, your guess is as good as mine.
I can't wait until this is implemented in Google camera.
As three years have passed already, is there a software that have implemented this feature automatically? (maybe with LiDAR data?)
It’s also called the parallax effect
Introducing the newly iPhone XX Max SX now with 3D Ken Burns Effect.
Do they mention which network theyre using to make those nice coarse depth maps?
Oh, I see, it lets you make Business Casual videos
[old-timey civil war-era voice, slow zoom over black and white image, cursive handwritten letter appears on screen]
"Dear Fellow Scholars... morale is low... it has been 3 months since we've had any hope of groundbreaking research... tell Mother to hold on to her papers for me."
And hopefully this technology will also lead to better false lens blurring effects from single images.
Kinda off topic but this is almost exactly what LSD looks like
rip lytro cameras they were rgbd cameras
Károly! Egy kérdés, Magyarországon laksz vagy valahol kint?
2:02 He had it coming
they need this in google street view, maybe the whole world can be explorable like a video game
In the future, every image will be one of those 360° things
What about lidar in the new ipads, Can we use that for training this model?
@4:30 There's gif's in the pdf files. "WHAT A TIME TO BE ALIVE!"
I just watch these videos to be called a fellow scholar
is there a program i can download to do get this effect
Someone really needs to make a pc/mac/ios/android app store for these papers so us plebs can have fun with them
I assume they just sell the tech patents..
0:44 and booom haircut :D
I wonder what happens if 3D depth map of high res pics, like Apple RAW from the newer iPhones gets used instead, since depth was captured at time of shooting.