How Games Have Worked for 30 Years to Do Less Work
ฝัง
- เผยแพร่เมื่อ 9 พ.ค. 2024
- We explore the evolution of culling and visibility determination in video games, building on work started over 30 years ago, and evolving with every generation of hardware.
Gamedev Courses: simondev.teachable.com/
Support me on Patreon: / simondevyt
Follow me on:
Instagram: / beer_and_code
Twitter: / iced_coffee_dev
In this video, I implement some of the various visibility techniques used by AAA games like Just Cause 2, Battlefield, and Splinter Cell, and others. We walk through the history of GDC presentations, and see how the techniques have evolved to what's considered state of the art today.
References talks:
advances.realtimerendering.co...
advances.realtimerendering.co...
gamma.cs.unc.edu/papers/docum...
developer.nvidia.com/gpugems/...
www.cs.princeton.edu/courses/...
blog.paavo.me/cities-skylines...
www.humus.name/Articles/Perss...
www.ea.com/frostbite/news/cul...
www.guerrilla-games.com/read/...
blog.selfshadow.com/publicati...
www.selfshadow.com/talks/rwc_...
aras-p.info/texts/D3D9GPUHack... - วิทยาศาสตร์และเทคโนโลยี
![[UNCUT] ฝันรักห้วงนิทรา | EP.1 (1/4)](http://i.ytimg.com/vi/g7F6i0ganRU/mqdefault.jpg)
Patrons can now vote for the next video! Thank you for your support.
Patreon: www.patreon.com/simondevyt
Courses: simondev.io
When DARPA s brain modem allows us to enter the matrix and fly like superman and experience snatching we want all this hard work on simulation rendering will really pay off.
Thank you for the video. I want to point out that after 17:03 the captions went nuts.
Your a very good teacher straight facts easy to understand and a calming voice..🎉
U r sick
I can tell you're a developer because you sound like you haven't slept in six years. That was an amazing explanation, so kudos on your tireless efforts.
The not-sleeping thing is more from my kids
Sounds like Jason from “Home Movies” lol
I can mostly tell from the name
yee i was gonna say john benjamin @@CrowdContr0l
Thats some bs stereotype for devs. U make shitty code if u dont sleep well
8:48 What's interesting here is that for the Portal64 project (Portal port to the N64) the dev decided to skip having a depth buffer and instead sort things from furthest away to closest using the CPU. The reason for this is that the N64 has extremely limited memory bandwidth and also more CPU cycles than it can use. So he'd rather spend CPU power sorting things than clear/write 32 bits of depth data to every pixel every frame. it does also help that the nature of Portals maps makes it really easy to cull things, anything that's not in the same room or in a room that's visible through a portal, open door or window can be culled without even thinking about it.
I don't think he mentioned this in his video, but given the game is running at 320x240 this presumably also saved him 150kb of memory. A not insignificant amount when he has 4mb total to work with.
Yep, it's called Painter's Algorithm, and it was commonly used in the era before depth buffers. The whole era before modern hardware is really cool, and needs it's own set of videos! heh
An important clarification here, the portal64 example is per display list, and not per triangle.
You should see what the community of 3D creators on Scratch have come up with to overcome the computational limitations of scratch it's wild stuff! Painters algorithm and BSP enhanced painters is common place!
Working on the N64 is weird in general. the rambus is so ludicrously slow compared to the rest of the system that you end up having to do a lot of stuff downright backwards compared even to contemporary systems.
It also makes me really appreciate modern development tools, because you look at the internals of games like Mario 64 and its pretty clear they were optimizing blind, without being able to actually measure what was taking the most time, or even whether certain changes were helpful or harmful.
Crazy, I've heard stories about the older architectures being super strange. I landed on the tail end of ps2, which was apparently a total nightmare to work with as well. Never knew any N64 people, but watching some of @KazeN64 videos makes it look super interesting.
I LOVED Prototype growing up. Super cool that I just stumbled across a dev on youtube. Your channel is great btw
The power creep was so fun, really made you feel like a trillion-dollar bio weapon
I've always wanted to find and thank a dev for Prototype. That game and the next were amazing and the ability to bring destruction to the town was amazing. Thank you all
that game made so many memories for me. thank you, Simon, and the entire team on Prototype.
@@courtneyricherson2728 I was a bit disappointed with the story direction of the 2nd game.
It really felt forced to make Mercer the villain instead of just apathetic. So they should have (in my OP) went with something or someone else as the villain.
I rarely do any game development, but love your content! It's good stuff. You and Acerola have become one of my favourites to watch and learn about how these digital worlds come about.
I love Acerola's content too!
I came here to say the same Acerola and Simon are the goats!
same. I think graphics programming has a lot to teach about programming in general, especially math and the performance of algorithms, and it intrinsically visualises the thing our program is manipulating, which naturally lends itself to clear educational content and a tight feedback loop for problem-solving and evaluating our methods
-but when I do… it’s Dos Equis.
@@arcalypse1101let's not forget Sebastian Lague
This video is insane. All the stuff I was looking for for years on the internet, just made available in a simple, condensed, purpose-made effective educational video with no fluff. Thank you so much. If only every teacher was this good (and every research paper was readable).
@@leeroyjenkins0 indeed, it builds really well onto foundations you gain from uni. Now the problem is just time. Even the doctor's degree genius programmers and the studios spent decades iteratively developing and adding these algorithms to the game engines with each new game. Building a new game engine just seems like such a daunting, massive task. And that's just the rendering side of things! You still gotta create tools on top of that to be able to work with your engine. Modifying existing game engines like UE might be the way to go (Deep Rock Galactic devs chose this route), but even then, you gotta know the engine pretty well, which is stupidly specialized knowledge, as well as know the algorithms involved.
Not to mention he got H John Benjamin to narrate the whole video
@@1InVader1 Building a new game engine isnt a daunting massive task, unless you want to compete with UE or Unity.
@@rykehuss3435 it only makes sense to make one if you want to do something that those others don't do as well (destruction, voxels...). If you want to do the same, or anything less, what's the point? UE already exists, made by people smarter than you, might as well use it.
@@1InVader1 I agree
I guess the reason why Cities Skylines 2 does not bother with occlusion culling is that in a top-down perspective there are simply not many objects behind each other (in contrast to e.g. a 3rd person game).
I feel like you'd still benefit heavily, depending on the shot. When the camera is overhead, the amount of "stuff" is naturally constrained to an extremely small area, thus your occlusion needs aren't high anyway, vs when the camera is lower.
But this is mostly conjecture, so take it for what it's worth.
@@simondev758 Its an interesting thing to think about, as a lot of simulation games have had this problem where once you zoom in to look at closer detail, performance tanked.
However its especially problematic in Skylines given its a game that encourages you to zoom in to street level, why else make it so detailed? Seems like a big goof not to realise that's going to be problematic.
I literally had the thoughts of "oh, so you can compare foreground and background objects in screen-space" and "you don't have a depth buffer yet, but you have the one from last frame" before those subjects came to be implemented in the demonstations. The examples were really well explained and very intuitive thanks to the visuals!
The speed at which it does all the calculations of what should be drawn and what shouldn't always blows my mind.
A lot of the HZB ones can be done in less than a couple ms.
That's because you're thinking of it as physical real world objects. When you think of it as data in a notepad file, all the computer is doing is reading very quickly.
I'm currently making a 2D game and it kind of blows my mind that the tradeoff of not drawing objects is worth the time it takes to check what should be culled every single frame. Surely simply checking which objects should be culled is a massive processing task
> Surely simply checking which objects should be culled is a massive processing task
worth it if drawing object is a lot more massive task
To be fair, if those objects have a simple geometry and only a texture, culling on single objects does cost way more than not. For example, in Minecraft you wouldn't want to run culling calculations for every single block. But as objects got more and more complex and shaders entered the picture, that shifted.
In a 2D game you don have to check any objects, just have your world in a grid (2d array), then the culling is just your for-loop which ony iterates through the part of the grid which is visible on the screen.
Well actually it would be worth it if drawing object is a lot more massive task than sorting _and_ drawing the remaining stuff, so the workload is actually got two parts to it @@RandomGeometryDashStuff
@@rosen8757 Thanks for this, I'll be implementing it in my engine.
That's a pretty darn great summary for beginners, did this for 3 years myself and it's definitely one of the more challenging yet fun programming fields there are!
P.s. i trawled through hundreds of pages of documentations and papers with no friendly youtube video to guide, you still can't avoid that really if you want to become actually good at this, but do watch the video to get an overview.
Hah, yeah if you could become an expert off a 20 minute youtube video, that'd be great. No shortcuts unfortunately.
You're a great teacher. There are only a handful of good youtube channels where you can actually digest the content. This video is gold.
Optimizing CPU performance is something I enjoy doing a lot, very interesting to see how optimizing GPU operations is done. Loved this!
Also makes me grateful for game engines which mostly do this for you already haha, not sure I'd want to do this from scratch unless I really needed to get extra frames
The fun part of being a graphics engineer is that you end up doing a tonne of both CPU and GPU optimization.
Glad I found this again because I wanted to let everyone know it's better than a bedtime story, I fell asleep watching and listening this and woke up to a dead phone with flat battery. Best sleep in ages.
I've watched it all this time and found it very interesting.
Your channel is an absolute gem, so many high quality videos about topics that are really hard to find online. Thanks.
This is definitely my new favorite of your videos. It's endlessly fascinating to me to hear about the crazy things that are done in rendering pipelines. I love the GDC presentations where they dig into the minutiae of their rendering customizations.
Prototype was hands down my favorite game when it came out, and years after. So many days coming home from a crappy shift to take out my frustration on the zombies/soldiers/citizens/mutants of New York. Thanks for the memories.
Prototype is one of my favorites games, both of them, I hope the source code gets leaked so we can get better mods, since it doesnt feel like we're getting a new one
i hope youll read this, because this video has really inspired me. not only do you explain things in a really easy to understand way, you also carve out a value system of things. "this is easy. not that complicated. not a mystery really" these really help to get a feel for the underlying relationship of all the different approaches (and systems) that you would only get in a 1on1 conversation. thank you for showing its possible! great video
Honestly a lot of gamedev isn't super complex, but presented in weirdly convoluted ways.
Hi Simon, I've been following you on TH-cam for a couple of years and I'm very inspired by computer graphics and game development, currently making a game on Phaser 3. Thank you for explaining interesting techniques and sharing your incredible experience, I learned a lot thanks to you
Ohhhh dev who worked on prototype teaching me game dev, i feel so bleesed. I absolutely loved that game, and still love it, thank you for putting in so much effirt behind it. And thank you for these amazing teaching, keep it up man, much love and respect.
As a mechanical engineer i like how you kept this simple yet technical in terms of explanation. This is a skill in itself. You got yourself a new subscriber! Keep it up my man
Explaining major breakthroughs in game industry for a given problem is so interesting! Thanks a lot Simon and keep up the good work!
You do a great job of explaining abstract concepts in a clear and concrete way, thank you.
I'm a dev myself and I gotta say, 90% of this was new info and the last 10% of the new info kinda flew over my head a little. This is amazing, thanks a lot. It's not often that we get to see the nitty gritty inside stuff that you don't directly work with.
Somehow you manage to "destress" me while teaching what could seem like a complex topic but you manage to break it down so it seems so simple. I like your javascript projects and I have converted some of them to typescript.
I read that as _distress_
There needs to be a VR game engine that only renders nearby objects twice (for the left & right eye) and far away objects once, because far away objects don't need stereoscopic vision. This would would save resources and improve performance.
That's pretty clever. I'm curious about how it would perform
Since distant objects are more likely to change less pixels per frame, you could render far away objects with a lower frame rate as billboards. And then if necessary smooth the movement by shifting the billboards in between their frames. Then, with the saved performance, maybe you could even render important objects at a higher resolution than they would be normally at such a distance and then downscale it instead of doing anti aliasing or something.
@@DeusExtra ah yes, that’s another good idea. Render far away objects at half the frame rate or use alternate eye rendering like the Luke Ross VR mod. But in VR, far away objects need to remain at high resolution because low res is very visible in VR. That’s actually the biggest immersion breaker in VR, when you can’t see far away objects, like in real life.
@@djp1234 meh, not seeing far away objects is perfectly immersive for anyone nearsighted. Lol
@@voxelfusion9894 how does nearsightedness translate into VR vision? The actual display panels are close to your eyes. And do corrective VR lenses help with that?
Subscribed. One of the best and more clear content about graphics programming that I ever seen in youtube
I actually loved prototype! I bought that game right when it released. I really wish they would have done more with the story, and made a good sequel. You guys really did do good work on that game. The free roam playability in the aspect like GTA, has been unmatched in that particular flavor of genre. Now I'm going to have to dust it off and see if I can get it to boot on something
A quick subscribe from me! I look forward to you going into transparency shenanigans. It surprises me that to this day it is not unlikely for a player to run across transparency issues. I remember even in the recent beautiful Armored Core 6 I found a situation where smoke consistently interacted badly with something. And in playing around with my own projects, I've gone overboard with "transparency is beautiful" too many times, and keep having to be mindful of performance impact.
am surprised to hear you worked on Prototype... am a car guy, so the none car games i played could be counted on one hand, Prototype was one of them that i loved so much, it was like a modded gta to the younger me, just a weird amazing experience really
Prototype was a fun, unique title, miss working on that team.
Thank you so much for sharing your knowledge with us, love it when you bring out new videos!
Loved the video, I did quite a bit of reprojection shenanigans about ten years ago with the DK1 and DK2 to improve perceived frame times for stuff outside of our foveal vision!
Man, you're the best. Rendering is an entire discipline unto itself and it's always on the horizon of my attention/knowledge. I finally found the time to study flow fields and I'm moving onto nav meshes soon. One of your first videos was on A*, I'd be interested to hear what you have to say about group pathfinding.
Immediately subscribed. That's a really good content and you have a nice voice even for a long videos.
This was really awesome! I used to develop for the nintendo DS, so learning to develop with very strict constraints was really part of the job.
This format with in-engine examples really set the video apart, excellent job man!
Thanks! I worked with a guy who was fresh off of DS years ago, very smart guy. That platform sounded like a pain to develop for.
god...just the sheer amount of knowledge and sublime ability to explain usually not quite so straight-forward concepts (when read black-on-white from a uni slide or a blog post written in a VERY dry fashion) *THAT FREAKIN WELL* just amazes me to the point that you Sir have officially becom my role-model (no simping intended). And I mean...duh, no wonder you were (or are, dunno) a Google sofware engineer, cuz that is the level I aspire to become anyhow one day.
Thank you A LOT and I hope this world blesses you and your fam for everything!
Super thankful that you make such amazing vids!
Cheers!
Awesome ! thanks a lot ! The explanationn coupled with the rendering are amazing !
Easy to follow, in depth explanation of some pretty complex concepts, waht more can you wish for... Thank you
One of the videos on this site that I enjoyed the most. Please make more!
Truly informative. This is a great perspective, i usually hear arm chair devs talk about game development. I learned a lot here. Subbed
Fantastic education, especially for a lone developer trying to learn more, who is unsatisfied with simplistic answers. These twenty minutes were more valuable to me than many hours of the Unity "tutorials" I have watched. Thanks for being so helpful.
Man, I've done Game dev for over a decade and this still sounds amazing :) Love your channel!
Really excellent video! I don't know much about graphics/rendering so I found this fascinating!
It's the cool but also exhausting thing about graphics, you basically have to retrain constantly hah!
Hey Simon, amazing video, I really love how you go in depth into the nitty gritty of optimization and the history of it. One such argument I'd love to hear about is collision engines, with broad, middle and narrow phases, aabb collisions, spacial partitioning, the challenges of long range ray and shape casting and so on, I feel like there are so many different interesting things to talk about in collision engines
Glad to hear you enjoyed it!
I'd love to dive into more optimization topics, but I think I'll leave collision engines out of it. I strongly dislike when people pass themselves off as experts in things that they're not, and I hold myself to that same standard. I haven't done anything beyond superficial physics work, so I don't feel especially qualified to talk about the subject.
I'd encourage you to find resources or experts that specialize in that area. Would love to send you in a direction, but I can't think of any offhand unfortunately.
I can imagine using the terrain or buildings in a city as occlusion objects has big benefits real quick.
This kind of quality of content is amazing as a graphics programmer to have access to. I'm amazed by your channel
Thanks! What do you work on?
As someone who never made it that far past the Painter's Algorithm section in any graphics book, this was great 🙂
Loved this, as an enterprise software engineer with no game development experience, I found this highly interesting and really easy to understand. You did an amazing job, thanks and you have another subscriber. 😁
Very nice presentation. I learnt a lot and got some inspiration. Thank you!
This is awesome. I have always been left scratching my head after reading papers and watching whatever few videos even exist about the more advanced culling methods here. Is there any chance you have a github with source or will you make a course covering each topic in detail with in depth code examples? I think people would buy it. Thanks for the video!
Thanks for your work on Prototype! Has a special place in my heart
WAIT YOU WORKED ON PROTOTYPE
THAT SO COOL I STILL LOVE THAT GAME
[Prototype] is one of my favorite games of all time. It was the first game I ever got a platinum trophy on. Thanks for working on it.
Just a small tip for visualization, I have trouble seeing a difference between Red and Green, which was worse with the Green and Yellow examples - and for those with Colorblindness, contrast is always the eye's first priority over color. Better to use colors that are complimentary, or better yet, just white and black for visualization.
(And yes, it's hard for us in Graphics Programming) :p
Really love your video! super simple and a great start to understanding graphics and optimization. Subscribed :3
I think someone else brought that up, and it never occurred to me. But I will 100% strive to be better in the future.
Here to second this, yellow is a devious colour that is seldom seen
@@simondev758 thanks for taking it seriously, I appreciate how you handle feedback on your videos
You can look for ready-made "accessible color palettes" to drop in, or keep it simple with contrast and patterns. It really does help, I kept pausing this video just to tell the effects apart, and the problem affects everybody when you have different calibration for every monitor.
Next they'll be hiring deaf people as lifeguards. What's the world coming to. 😂
That last "state-of-the-art" demonstration is so cool! I honestly never even realised it was common to do visibility culling outside of precomputed visibility structures. But not only is it done, there's some very interesting algorithms to lean on. I especially love algorithms that don't rely on temporal reprojection, so that last one (use objects visible in the last frame as occluders) is quite fascinating to me.
Mr.doob is one of your patrons! Actually I’m not surprised. GG
Also just noticed. Too cool
I am not graphic engineer, but the content you make is extremely interesting to watch. Thank you for your work sir
What learned me a lot of optimization was actually the source engine when it came to mapping in Hammer. Now using your video to learn more about other types of optimizations that might be possible for Source or at least in Hammer. Thanks for sharing this type of content and information to anyone that wants it!
I am also going to code my own game that is heavily inspired by a game that is poorly optimized. So watching these will hopefully ensure I will not make the same mistakes those developers did.
Thank you for a great explanation! If you are suddenly out of ideas how to proceed, I am personally very interested in how this process works with shadows, lights, reflections etc from the offscreen objects. By playing Hogwarts Legacy, which is an UE4-based game, I've noticed that some elements, like reflections in lakes, are often suddenly popping up when I slightly move the camera, which causes an unpleasant experience.
Yeah, one of the options in the last poll on Patreon was exactly that, the various reflection techniques used over the years, and the tradeoffs between them. I haven't played Hogwarts Legacy, but it sounds like SSR, or screenspace reflections, which is what I showed part way through this video (the short of the cheap reflections on water when I talked about the depth buffer). It's a limitation of the approach, but it's super cheap.
great summary, thank you for doing work to condense all these techniques into easy to grasp visuals.
Nice video. I had once read a paper on culling strategies. Its just amazing how smart people are in the gaming industry. Some of the most amazing algorithms came from the gaming industry.
So if a tree falls in the woods and no one is around to see it...
Great video as always! Thanks for the incredible overview. I've been wondering about occlusion culling for a long time as I was thinking of implementing it in my toy engine and now I finally feel confident about how to do it properly. Sadly it seems like a lot of work implement the two pass system so I probably will never do it, lol.
It's actually not, you can bang out an implementation in a few hours
I am enlightened wow!
I am really into game dev and graphics and the topics you ve pointed in the video are amazing i ve learned so many things
I never worked with game development, but I love this channel. I love to hear about clever solutions to optimization problems. This video was particularly interesting.
I have always been fascinated as to how games were rendered on screen. Great video.
Cool stuff. I've sometimes wondered how games do this and avoid unnecessary work. Thanks for sharing
Thanks man for this informative video. You are doing great job
It's like if Bob's Burgers explained Computer Science
I appreciate that you go straight to the topic
That is really cool you worked on Prototype. I really enjoyed that game.
For me, the most interesting thing is, how the "MOV AX,CS" and similar lines became an open world game with theoretically unlimited gaming time.
40-50 years ago one or two people worked on a project (code, music, gfx was the same person sometimes) and then, 20-25 years ago the games became so complex that some pc games had longer credits list than a Hollywood movie.
I didn't know you worked on Prototype. I love that game!
You worked on prototype ?
Bro it was my favorite game it was unique and amazing graphics and also amazing gameplay feeling. Wow.
Woah! 😮 Crazy you worked on Prototype. I loved that game, and I have always remembered it from time to time. Maybe you could make a video about your work on the game.
Would you be interested in that? Brief summary: I did some of the early r&d for the graphics, did a big presentation at GDC, and during production mostly did backend and optimization work.
@@simondev758Definitely should do a video about it. It would be a great video to watch.
I just discovered a new excellent channel, then looked at your channel, and I found out I had already watched a few of your videos.
Off topic but those trees in your demo is beautiful! I really like the style of them!
really appreciated this, and good visualizations to go with the explanation. thank you!
Glad you enjoyed it!
I was waiting so long for you to bring up temporal cohesion.
Great video. I'm a 3dArtist, not a programmer of any sort and there might have been a simple explanation for it that I've missed but how does culling account for things like long shadows or global illumination, that leaks from off-screen into the visible scene? ...Maybe worth a part2? :)
Excellent video, so interesting and clearly explained. Nice work. And you made me reinstall Prototype (at least trying to ... ).
This is information rich / dense yet so simple. thanks!
Prototype was one of my favorite games back in the day. Very cool to hear about it here.
amazing video. Your expertise are super evident
Never learned about HBZ, i just knew i had a depth buffer i could access.
Great video! Well done at explaining how it works!
Don't know how i will use this information atm, but I'm sure it will help me one day down the line.
Also now I have lot's of questions in my head how this might be shared with both eyes for VR.
Back in the early-mid 2000s I remember learning a bit about bsp (binary space partition) maps for ID tech-based game engines. Maps were compiled ahead of time to determine what parts of the map would be rendered depending on where the player was. It's neat to see that most of that process is done in near real time now.
Super interesting stuff, but I gotta say as we got to the more complex stuff, it started to fly over my head a little bit. You had great visual illustrations for the simpler stuff, but not for the more complex stuff. Probably would be more difficult to make, but would certainly help! One example of where I got lost was the hierarchy tree stuff.
Gran Turismo has used this technique for at least 2 decades. WipEout 2048 also used this technique on PS Vita, but if you do some funky things with the photomode camera, you can see loads of massive black boxes through large chunks of level geometry labeled in bold red font "Blocker."
You worked on Prototype? That's one of my favorite games!
The video is very informative, and relatively easy to understand even to someone who knows nothing about game development, though there's a minor issue with captions. At a few points the captions are mismatched, like at 12:10, at it takes a few seconds for them to catch up.
Ah ok I'll double check the captions.
Interesting and educational. Thanks Simon!
i love your content so much simon, all of it its amazing
I don't know why I watched this whole thing and took notes. I don't work at all in this field, but it's so interesting. THanks!
I was thinking long about how to deal with not just culling, but ray tracing, collisions and gravity simulation too, for a space game.
and yeah, cache optimization is important, but so are tree structures, esp for stuff where everything interacts with everything.
I want to do a hybrid approach, where the tree nodes serve as containers (indexes) for arrays of properties.
I'm super excited for it!!!!
but for now i gotta work on simpler games so i can make a name for my self, and make it as a game developer \o/
You worked on prototype??? Bro that is like my favourite game! Keep it up
Great video, I wish it was made a few years ago :-) In my occlusion culling journey, I originally took the HiZ approach on the GPU which worked out great at first. It became a problem though when I wanted to do more with the visible entities. I tried in several ways to send data back to the CPU but there's just too much latency and not enough types of indirect commands to keep it on the GPU, so I went the CPU route instead. Intel has a great paper for their method of CPU-side HiZ implementation, "Masked Software Occlusion Culling". They also provide an open source implementation which has performed well for my application.
Yeah I wanted to call out to Intel's library at some point, but didn't have a good reason to.
This was very informative. I'm not really involved in any game dev. Thank you!
Thanks for sharing this! Very interesting topic!
I dont know much about this stuff, but this was really understandable, great explanation
What an elegant way to solve the depth buffer dependency issue. Render the simpler version of the view to extract depth data and then render the high resolution view.