Hey mr Ronald Weasley, I just want you to know that we appreciate your dedication to this channel and we in no way judge you for not making videos as often! You're the reason I got into C++ a year ago and now getting close to releasing my first game!
@@TheCherno Don't know if this will be helpful, but in my experience writing compute code for GPU's (mostly with OpenCL) it is invalid memory access which causes the worst issues (e.g. accessing non-existent array index). AMD cards will often run without reporting issues but Nvidia cards will crash or refuse to even run the code (which is good because it can help prevent damaging the GPU).
I hope you read this cherno. You should realize that your cpp videos have gotten 1000's of people into programming and c++, including me. That means a lot of good softwares made by those people wouldn't have even existed if not for your awesome videos! Your impact is large! Keep doing this we love it
in my last year of uni working very hard and feeling depressed even though I know I'm on track, I appreciate you making this video, reminds me that even people as skilled as you face the same struggles as everyone else!
The key is to "give up" in a smarter way, in various occasions I had problems I couldn't solve that I thought I wouldn't be able to and after days of trying and failing I "give up"... but not forever!! What I do is that I revisit the issue months later some times even a year or more later, and to this day I haven't had a single thing I "couldn't" solve. We don't know what we don't know so sometimes just having time go by while learning other stuff may trigger the thing that unlocks you to that old issue. I constantly have 1 or 2 of these with me, in fact just about a month ago I cleared one of those. This proves to me that anyone can do anything given the right conditions, just don't give up those things and instead think about them as challenges yet to conquer.
I take a similar approach when learning gamedev, I'm learning to make games in python using the pygame library and I always tried to implement my own particle system.. but was never able to and I don't know why.. I decided to give up and do other stuff while hardcoding the particles in every prototype (not a single game released but I will participate in my first Ludum Dare in 3 days), after 1/2 month I decided to make my own framework to speed up developement, and when I finally convinced my self that I needed to make the particles.. it workedat the first try.. looking back I did learn some more about classes and maybe that helped, and now I can't even figure out why I couldn't do it before..
That's what I do too. When I'm really getting roadblocked I shelve it for a while and come back later, clear headed and refreshed, and with more energy to go through the hassles of brainstorming and researching. It pays off to give your brain some time to process information, even if subconsciously. It's like working out, really: you have to allow your muscles some rest to maximize your gains. The brain also needs time to get organized. Doing unrelated stuff every day and being well rested helps too.
Potential list of common bugs: a) synchronization - barriers, threading, etc. (check VK barriers and disable multi-threading to see if the issue goes away) b) resource indexing - check if you properly copy the memory of the cube maps. A lot of people make subresource indexing bugs, etc. c) check the constants and uniform buffers you are sending to the GPU to see if you are always sending proper values (use some asserts if some values become unexpected, large, negative, etc) d) check shaders if you have some dynamic indexing going on. If you send in wrong values, you may go out of bounds and access invalid GPU memory. e) check descriptor set management if you are doing something fancy with it, or somehow indexing into it (bindless rendering).
I am 100% with you on that one Cherno, sometimes you definitely need to take a break and do something else while your brain quitely comes up with solutions to problems in the background, I can't tell you the number of times ideas for fixes have just popped in my head when I was driving to Uni or taking a shower and I was like "ohh, that could work, I can't wait to try this!!"
Haha... my last crash was solved when I woke up in the middle of the night with the answer. Can stare at code for hours and not see the bug. Go away, clear you mind and it's amazing how often it will be the 1st thing you see.
7:30 I had a very similar issue with gl when I was writing my test game engine years back, iirc I had a missing semicolon in a headerfile, compiler never complained but leaked somewhere bizarrely with super strange behavior like you are describing.. Took me ages to figure it out, i know your post is old so irrelevant now but for posterity I think I got it with valgrind and a lot of removing code parts one by one.
I'm working a a single project since 11 years now. Still fun but sometimes you do great, sometimes you are simply trying to move a wall. Sometimes problems will drain all your energy for a bit. You set a massive goal in front of you. The only thing to really help is a team.
Im am still writing on my game engine and I started right with your series 2 years ago. My engine is not so far as yours because I decided to write my own math library but I know this feeling so well of just wanting to give up! Most of the time when this feeling comes up I take a break for a month or two to get excited again about the project and possibilities you have with your own engine!
16:10 There should be a button to match the directional light's direction to the sun, otherwise it's really confusing if you set up the sky for the first time. Looking forward to seeing more of this! :)
I'm sure you'll figure it out, sometimes you gotta take a break or even take a step back and find a new way when getting stuck. but hey you're already so amazing doing all this engine stuff on your own (solo for the longest time) you got it cherno :D
Haven't watched any of your videos in a long time, but I wanna say: you inspired me to get more into C++ and computer graphics and I actually recently finished courses related to both :)
Had a tonne of hige bugs I had to deal with in my game Saturn 7 but each time I slept on it and my brain often came up with a solution to try the next day...amazing how your brain never stops...take a break as needed and dont feel too much pressure. Enjoy each little success along the way and before long the thing will be done as expected.
You've done some great work. I remember last year you were full of a lot of expectations for Hazel and it looks like they're finally becoming reality. You did a great job. Don't underestimate how much work you put in.
its important to stay motivated! solving those crashes seems not like much fun. it is definitely ok to take breaks. you will eventually fix it :D keep it up!
hey Yan, I just wanted to say you are doing a brilliant job. I started watching your C++ vids about 3 months ago, being a c# and java developer your tutorials was exactly what I needed, you inspired me to learn more about Graphics APIs, So much that I started a Vulkan course, I'm not gonna lie, it's hard but I've rendered my first 3D cube and I'm very Proud of it. Thank you for this video, it reminded me that there will be hard times where we feel like giving up but that doesn't mean we should. I wish you all the luck to fix that bug with Vulkan, if I where more experienced I would definitely have loved to help out, maybe one day
I used to have issues with environment maps in my Vulkan renderer. It would be more frequent the larger the environment map was. So a 512x512 cubemap was less likely to crash compared to a 1024x1024 cubemap. It was also more likely on laptops when they weren't in performance mode. I reckoned it had to do with the vulkan device timing out
I've gotten a similar error when working with DirectX 11. If the compute shader is too complex the driver will catch that the shader is taking too long and assume an error happened in the code to make sure the GPU does not get stuck. A way to fix this is to either lower the load put onto each call of the compute shader, or watch for the error to occur and recreate the device so the program does not crash.
@@lazergenix I prefer lowering the load by decreasing the amount of work needed to work with an environment. Especially when it comes to IBL. The environment map for the most part doesn't need to be thaaat detailed for it to be effective. The benefits of working with floating point images
Few ideas, though I've never worked explicitly on Vulkan issues: - it sounds like some internal state to the API is getting confused - have you tried stepping through the disassembly? - What clues can you find about what memory Vulkan is using, and what its referencing when it crashes? - How does its contents change when it works vs when it doesnt? Can you dump Vulkan's stateful memory in either case and diff them? - Can you provoke the crash in a minimal program? - If not, can you turn off other systems in your engine until the bug doesnt occur? What systems, when present, exacerbate the issue? I'm sure you've tried some or all of this. For inspiration on fixing these kinds of issues I watch the talk "Forensic Debugging" on GDC by Elan R
Hi, have to say I totally agree. There are times when I too get fed up with something and I just divert off and add something "shiny" to the software, as you say a self reward. It really does help.
@みやぞのさくら Its great feat that they managed to make such a small game engine, but too many games I've played from itch made in godot run like shit on my laptop. The performance of godot is so abysmally bad.
So in terms of the Vulkan crash, when I'd get the device lost error, this was typically triggered in the destruct as the device is used to clean up resources, but the device was being used still wherever the crash occurred, so the ACTUAL problem is being masked. I'm curious which validation layers do you have enabled? All? Verbose? Also, have you tested this in RenderDoc? it spits out the validation layer errors as well, I've caught a few bugs that way as well.
Cherno, have you tried using NVidia NSight or RenderDoc to debug your GPU crash? I have used those before to debug and root cause DirectX crashes and would imagine they should help with Vulkan as well. If I were to guess, I imagine you might have an issue with data alignment, data packing, or accessing out-of-bounds memory in your Compute Shaders. Since also Vulkan gives you control over synchronization, I would double check your memory barriers as well (a buffer may contain incomplete data). Hope you figure it out. Cheers.
Somehow related, as a totally beginner I wanted to start making my own game engine with DirectX because I didn't like OpenGL that much so I decided to pick Frank Luna's book about DX12 In the beginning I was understanding what was going on but now that I've reached the second half of the book, I'm starting to feel very confused and tired of creating a lot of things, separating them only to implement one little thing. Now I feel you when you dislike low level graphics programming, especially when I see some cool effects that people create in few minutes/hours in Unity or Unreal But I still like DirectX and I still don't want to give up If someone is an DirectX expert and/or has read Frank Luna's books, please write me back
A error that occures randomly sounds like a race condition to me. You should probably check stuff like pipeline barriers and in general resource related things. Since the problem seems to be specific to the environment map code there may be something that is unique to that part of code. Sorry for any spelling mistakes :-/
I've been going about making engines for a few years now, it went from a quad to now an ecs. With that bit of experience I can say yes, game engines are really hard to make.
I am a Game Engine programmer myself, but I don't know a lot about rendering (I focus mainly on gameplay programming and tools). It would be cool if you made a video specifically about the various rendering APIs and what are the common points that could be exploted to write an API-indepentend renderer. I know a bit about oldschool OpenGL and some variants like EGL and NitroGL (from my good old PS3 and Nintendo DS days) but I don't know many of the most recent technologies and I could use some videos to wrap my head around those! Thanks and awesome video as always ^^
Have you heard about BGFX? I saw lots of games use it, like Minecraft: Bedrock Edition. It might get out of learning specific APIs but It really makes our lives more enjoyable.
Also there is GFX-HAL, It is good that we have open source low level cross platform rendering APIs, Personally I am currently learning Apple’s Metal API which is a nice API for learning low level rendering but when it comes to making a product I am definitely gonna use one of those cross platform APIs as I can definitely not learn all APIs for each platform and if it is good then why use specific API for each platform if we already have a good HAL (Hardware Abstraction Layer) rendering API
Rule #1.... it's always the changes that will effect nothing that will have subtle side-effects that will take weeks to track down. (As I have found out many times) Managed to get animated, instanced meshes working in mine now with decent lighting. ImGui was a turning point as being able to change stuff on-the-fly and having a nice interface for examining all loaded assets led to a lot of huge performance tweaks. Got loading all my models from like 4-5 seconds in debug mode to almost instant mainly by using a fast base64 decoder on the vertex arrays. I was debating a full binary file format but they can be hard to debug and look at with a text editor. Most of the file is human readable except for blobs of base64 on a single line for the mesh data.
I'm always a bit skeptical of these thicker api wrappers for most anything where it would have any meaningful benefit. The reason being: for where it is similar it probably isn't too complicated, and where it is different the specialized apis are probably much more suited syntactically to the task of expressing what they are, something the generalizing thick wrapper must accomplish in a balancing act between the apis it abstracts. This obfuscates the important bits and trivializes the easy parts even further. I don't mind me some magical code here and there, but there is such a thing as too much magic.
About the Vulkan crash, I don't know if you've tried enabling GPU-Assisted Validation or Synchronization Validation? Those can be enabled with vkconfig or in source code. I've found many bugs by enabling them.
I know you've only got Nvidia hardware, but it might be helpful to run the open source RADV driver for an AMD card on Linux and see whether it also crashes. There are things like shader trap handlers which may give you an idea of the issue, but even if you can't it's a high quality driver which you can ask for help with. If you can't get it building for Linux you could probably just use Wine. Unfortunately I've lent my desktop with a RX580 to a friend, and only have my Nvidia work laptop with me now. Another thing 1: Have you played around with validation layers? Another thing 2: vkTrace/gfxreconstruct. That way you could send a trace of the commands to people to replay and potentially find the issue without having to download and compile your project themselves.
I can very much relate to giving critical bugs some space and working on something that doesn't make you hate your life for a bit. I hadnt even realised it till you mentioned it; i'm currently writing my dissertation and that experience definitely going in there somewhere.
I have to admit I've liked your videos, since I'm making my own engine in C and OpenGL. I hit somewhat of a brick wall about cascading shadow maps, since you mentioned doing a video about that in more detail about how to do such a thing, so I will put my vote in to that if possible. ;) I have a large environment (It's a space combat game on planetary surfaces) so I'm looking for how to do a large area, and yet still have it look good when the player flies close to the surface and would see their own shadow or that of buildings they are going past. I also have dynamic lighting from the weapons and explosions, etc, so I'm not sure it's practical to add shadows to those light sources, but I would be happy with just the sun doing it. In anycase, I know you have your own series of things you need to do, and my problems aren't yours. But you wanted to know what people wanted to see, so I mentioned my vote. Thanks again for doing these videos!
It's actually kinda cathartic to know that skilled professional devs also get bogged down and frustrated by bugs like that too. And I totally identify with the urge to just work on something else for a while. I think for me it's primarily that I need to work on something that will distract me and give me a quick path back to the feeling of winning, competence, and actually accomplishing something after hours/days/weeks of banging my head against something that I just can't seem to crack. Once I've built back up some confidence and motivation, I can eventually dig back into the frustrating stuff again. I recently had one that had me feeling like I was on the verge of just throwing up my hands and quitting programming altogether. After weeks of frustration I finally took a break from it and worked on new shiny features for a few days. Then when I attacked it again, I finally stumbled across the root cause. It was a gstreamer bug specific to its WASAPI backend that was breaking drag and drop functionality on Windows in Qt and causing crashes because it was randomly deleting OLE objects that didn't belong to it. Not exactly the place I'd go looking for a bug that looked like a UI issue. heh
I also want to program a game engine which can provide an immense cgi quality graphics in conventional system like ps4 xb1 and entry level gpus just like cryengine and rockstar rage engine ithink graphics programming is a emerging environment in software engineering I think its more great and challenging than basic software development through this we can actually understand a true concepts of physics
I've just wanted to ask for more videos teaching graphics programming. There is OpenGL series, Vulkan and directX should be covered too. There is lack of this sort of content on youtube. We want to better understand graphics and it would be great to learn from You Cherno.
I really recommend you to integrate Nvidia Aftermath SDK into your engine, because otherwise you're blind to crash reasons. Aftermath is a must for page faults and other hard crash debugging.
Hi dude! Usually when I get those Vulkan crash on my side is usually related to wrong memory access (uninitialized or not mapped texture areas or shaders trying to use UVs in an unitialized area memory)
Yeah just do something else to avoid burnout! Something simple and creative is the best way. I got stuck on a project because I was so fixated on solving a bug without considering taking a break to clear my head. And that led to a terrible burnout situation emotionally and physically which in turn made the project took much longer time than it supposed to. Lol, I earned my lesson the hard! Always take a break and start something entirely new, simple and fun, to help the brain get excited again. Renew your mind as often as you can!
do not quit. I warn you. don't quit. this is a threat from a game dev who believes you have got some major talent even if it is a group project, this is too amazing to die. its almost looking like a unity clone. this is a good thing as unity works very well. fuck ue4 it sucks. I would love to start using you're engine when dx12 has support. please do work on all major versions of direct x 9+ ect as this will give it some major play in the game dev community. I cannot express how glad I am that you have this project going. more work = more likely your gonna get attention from other devs who would love this.
Hey, I wanted to know, when r u planning to make a tutorial on this aspect, I would like to know how u r able to choose backend at runtime Since almost all features like textures, window, framebuffers etc are specific to the backend
Maybe I could be of help with your vulkan crash. The debugging ability is a real issue with vulkan unfortunately even with all the validation layers and the driver logs. But after years of writing my own engine and doing very weird stuff using advanced vulkan API calls to the extreme, we end up with a brain that can have a very good understanding of what’s happening under the hood in the driver and what makes it crash. Also, it is possible that your current crash is a driver bug (I have found quite a few in the past two years especially on the Ray Tracing features and I got them to fix it) But if you’re using old Vulkan 1.1 very standard features and you are using latest validation layers, it’s most likely a bug in your code, or in the way your engine runs. When going from OpenGL to Vulkan, an aspect that can bite you easily is memory management and more precisely deleting a buffer or image that is currently in use by a command buffer that has not yet finished, this WILL cause a DEVICE_LOST error. This may seem quite obvious but we never know maybe you didn’t think of it and it’s what’s happening to your engine: This kind of bug is often caused by the fact that you may be running with two frames in flight at the same time so you should delete objects at least two frames AFTER the last frame that used it. Let me know if this helps, or you may contact me and maybe I can try to help. I have never seen the source code of your engine tho.
The game X4 had a lot of issues with that error in combination with all 3000 series GPUs. There was a hotfix from Nvidia ( GeForce Hotfix Driver 461.33 (Released 1/19/21)), since then it runs fine for me.
I fully understand that such issues can cost a tremendous amount of time and nerves, but we all know, the next time you will eventually will solve that issue - either by finding the cause or by changing the way how cubemaps are loaded. So keep going and do more creative things, when bugs stops you. I do this in my personal projects all time and most of the time, it helps me getting a different perspective. Speaking of personal projects, i am in the process of writing a multi renderer myself too, which at the moment is at OpenGL 2.1 level and i have no experience with vulkan. So i would be really interested how to abstract the video apis in the simplest way possible. In my current project which simulates fluid particles i have very simple rendering, except the fluid rendering technique which uses SSF (Screen Space Fluid Rendering). But compared to any render technique you have written in hazel, SSF is childs play :D
Yea something similar happend to me, I made a 3d interactive walk with compute shaders effect and they run differently on my new GPU ( less particless ) than they were on my old GPU. So I got new GPU and it looked worse. Now I am avoiding compute shaders xD grrrr. Hope I will fix that bug someday xD findout what going on.
![ประสบการณ์ จ้าง Developer ที่ผมจะไม่มีวันลืม [ภาค2]](http://i.ytimg.com/vi/pZbsSDRrmXA/mqdefault.jpg)
Sorry for the re-upload, video didn't export properly the first time :(
Please make videos on assembly 🙏
Ah, that explains it :)
do you have any tips or thoughts about work ethic with programming
Yes it doesnt in second as well because auto English subtitles is not loaded lol anyway have a nice day 😘
do you planning add java support for scripting?
Hey mr Ronald Weasley, I just want you to know that we appreciate your dedication to this channel and we in no way judge you for not making videos as often! You're the reason I got into C++ a year ago and now getting close to releasing my first game!
Nice
Could u paste link to it :>
What library's did u use?
“Ronald Weasley” 😂
He can sing, he can code..... Ed Sheeran is really something
😅
Let's not forget how great he was in Harry Potter as a kid. Guy's got talent.
@@davidmurphy563 Lmao!
hahahahaha
I read Ed Sheeran is a techophone. Hates tech.
sometimes when debugging blind, there's always the "comment out huge blocks of code and see if it still crashes" approach!
The problem is when it crashes 20 minutes into running, 10% of the time.
@@YoloMonstaaa always fun! ofc there tools like smartheap and valgrind you can build in to try to track memory corruption
Those work with gpu memory? My goto debugging tool is rr but I don't think that supports vulkan.
Yep, I've been doing that a lot - that's how I know it's the environment cubemap compute stuff :)
@@TheCherno Don't know if this will be helpful, but in my experience writing compute code for GPU's (mostly with OpenCL) it is invalid memory access which causes the worst issues (e.g. accessing non-existent array index). AMD cards will often run without reporting issues but Nvidia cards will crash or refuse to even run the code (which is good because it can help prevent damaging the GPU).
I hope you read this cherno.
You should realize that your cpp videos have gotten 1000's of people into programming and c++, including me. That means a lot of good softwares made by those people wouldn't have even existed if not for your awesome videos! Your impact is large! Keep doing this we love it
in my last year of uni working very hard and feeling depressed even though I know I'm on track, I appreciate you making this video, reminds me that even people as skilled as you face the same struggles as everyone else!
At least you didn't graduate right when the pandemic started.. I still haven't found a job as an engineer grad too
Best of luck. You can do it!
The key is to "give up" in a smarter way, in various occasions I had problems I couldn't solve that I thought I wouldn't be able to and after days of trying and failing I "give up"... but not forever!! What I do is that I revisit the issue months later some times even a year or more later, and to this day I haven't had a single thing I "couldn't" solve. We don't know what we don't know so sometimes just having time go by while learning other stuff may trigger the thing that unlocks you to that old issue. I constantly have 1 or 2 of these with me, in fact just about a month ago I cleared one of those. This proves to me that anyone can do anything given the right conditions, just don't give up those things and instead think about them as challenges yet to conquer.
Thank you for sharing this perspective! It resonates a lot with me
I take a similar approach when learning gamedev, I'm learning to make games in python using the pygame library and I always tried to implement my own particle system.. but was never able to and I don't know why.. I decided to give up and do other stuff while hardcoding the particles in every prototype (not a single game released but I will participate in my first Ludum Dare in 3 days), after 1/2 month I decided to make my own framework to speed up developement, and when I finally convinced my self that I needed to make the particles.. it workedat the first try.. looking back I did learn some more about classes and maybe that helped, and now I can't even figure out why I couldn't do it before..
That's what I do too. When I'm really getting roadblocked I shelve it for a while and come back later, clear headed and refreshed, and with more energy to go through the hassles of brainstorming and researching. It pays off to give your brain some time to process information, even if subconsciously. It's like working out, really: you have to allow your muscles some rest to maximize your gains. The brain also needs time to get organized. Doing unrelated stuff every day and being well rested helps too.
Potential list of common bugs:
a) synchronization - barriers, threading, etc. (check VK barriers and disable multi-threading to see if the issue goes away)
b) resource indexing - check if you properly copy the memory of the cube maps. A lot of people make subresource indexing bugs, etc.
c) check the constants and uniform buffers you are sending to the GPU to see if you are always sending proper values (use some asserts if some values become unexpected, large, negative, etc)
d) check shaders if you have some dynamic indexing going on. If you send in wrong values, you may go out of bounds and access invalid GPU memory.
e) check descriptor set management if you are doing something fancy with it, or somehow indexing into it (bindless rendering).
Mad respect for you for continuing this project for so long, I've never seen a any other youtuber pulling such a feat so far.
If you want to see one, have a look at OJReeves building a C# meterpreter from scratch basically
javidx9’s pixel game engine is quite involved too. excellent series to watch.
Wow, thanks for the suggestions guys, those series look super interesting
He needs to see this! Dont give up cherno, you are the inspiration to many many of us
Do you know chili?
I am 100% with you on that one Cherno, sometimes you definitely need to take a break and do something else while your brain quitely comes up with solutions to problems in the background, I can't tell you the number of times ideas for fixes have just popped in my head when I was driving to Uni or taking a shower and I was like "ohh, that could work, I can't wait to try this!!"
Haha... my last crash was solved when I woke up in the middle of the night with the answer. Can stare at code for hours and not see the bug. Go away, clear you mind and it's amazing how often it will be the 1st thing you see.
@@TheWhitde so true !
7:30 I had a very similar issue with gl when I was writing my test game engine years back, iirc I had a missing semicolon in a headerfile, compiler never complained but leaked somewhere bizarrely with super strange behavior like you are describing.. Took me ages to figure it out, i know your post is old so irrelevant now but for posterity I think I got it with valgrind and a lot of removing code parts one by one.
I'm working a a single project since 11 years now. Still fun but sometimes you do great, sometimes you are simply trying to move a wall. Sometimes problems will drain all your energy for a bit. You set a massive goal in front of you. The only thing to really help is a team.
Im am still writing on my game engine and I started right with your series 2 years ago. My engine is not so far as yours because I decided to write my own math library but I know this feeling so well of just wanting to give up! Most of the time when this feeling comes up I take a break for a month or two to get excited again about the project and possibilities you have with your own engine!
16:10 There should be a button to match the directional light's direction to the sun, otherwise it's really confusing if you set up the sky for the first time.
Looking forward to seeing more of this! :)
The UI all looks so clean!!! Looks great so far, it'll be really fun when you get to the stage of making Showreels/Tech Demos!
True.
Also nice hair. Looking fresh.
Amazing progress on Hazel. Really looking forward to seeing some architectural side stuff too if you do go ahead with making a video in that area
I want a mini-series on CMake!
Like 5 minutes ago i finished setting up cpp cmake for sdl2 opengl and additionaly first time imgui. I would love to better know what I ve done there.
Me too
I don't know if he uses Cmake. For Hazel he preferred premake.
"Nobody actually understands cmake; we all copy and modify snippets originating from the cmake primordial soup." - Some guy on Twitter
I'm sure you'll figure it out, sometimes you gotta take a break or even take a step back and find a new way when getting stuck.
but hey you're already so amazing doing all this engine stuff on your own (solo for the longest time) you got it cherno :D
Haven't watched any of your videos in a long time, but I wanna say: you inspired me to get more into C++ and computer graphics and I actually recently finished courses related to both :)
Had a tonne of hige bugs I had to deal with in my game Saturn 7 but each time I slept on it and my brain often came up with a solution to try the next day...amazing how your brain never stops...take a break as needed and dont feel too much pressure. Enjoy each little success along the way and before long the thing will be done as expected.
You've done some great work. I remember last year you were full of a lot of expectations for Hazel and it looks like they're finally becoming reality.
You did a great job. Don't underestimate how much work you put in.
its important to stay motivated! solving those crashes seems not like much fun. it is definitely ok to take breaks. you will eventually fix it :D keep it up!
hey Yan, I just wanted to say you are doing a brilliant job. I started watching your C++ vids about 3 months ago, being a c# and java developer your tutorials was exactly what I needed, you inspired me to learn more about Graphics APIs, So much that I started a Vulkan course, I'm not gonna lie, it's hard but I've rendered my first 3D cube and I'm very Proud of it. Thank you for this video, it reminded me that there will be hard times where we feel like giving up but that doesn't mean we should. I wish you all the luck to fix that bug with Vulkan, if I where more experienced I would definitely have loved to help out, maybe one day
Love the game engine series! I'm interested in how you would tackle a tear off pane for an additional viewport.
I used to have issues with environment maps in my Vulkan renderer. It would be more frequent the larger the environment map was. So a 512x512 cubemap was less likely to crash compared to a 1024x1024 cubemap. It was also more likely on laptops when they weren't in performance mode. I reckoned it had to do with the vulkan device timing out
I've gotten a similar error when working with DirectX 11.
If the compute shader is too complex the driver will catch that the shader is taking too long and assume an error happened in the code to make sure the GPU does not get stuck.
A way to fix this is to either lower the load put onto each call of the compute shader, or watch for the error to occur and recreate the device so the program does not crash.
@@lazergenix I prefer lowering the load by decreasing the amount of work needed to work with an environment. Especially when it comes to IBL. The environment map for the most part doesn't need to be thaaat detailed for it to be effective. The benefits of working with floating point images
Few ideas, though I've never worked explicitly on Vulkan issues:
- it sounds like some internal state to the API is getting confused
- have you tried stepping through the disassembly?
- What clues can you find about what memory Vulkan is using, and what its referencing when it crashes?
- How does its contents change when it works vs when it doesnt? Can you dump Vulkan's stateful memory in either case and diff them?
- Can you provoke the crash in a minimal program?
- If not, can you turn off other systems in your engine until the bug doesnt occur? What systems, when present, exacerbate the issue?
I'm sure you've tried some or all of this. For inspiration on fixing these kinds of issues I watch the talk "Forensic Debugging" on GDC by Elan R
Hi, have to say I totally agree. There are times when I too get fed up with something and I just divert off and add something "shiny" to the software, as you say a self reward. It really does help.
You are the best developer I've seen, don't ever doubt yourself
I would like to see Cherno use some available game engines like GODOT, Unreal Engine and share features he would like to have in Hazel.
Yes, something like an engine review. I'd watch that
Yes game engine code review would be awesome. I wanna know what he thinks of the godot code base
It takes time to effectively evaluate an engine, if you don't use it daily at most he'd discuss is surface differences that don't really matter
pretty sure Cherno had exposure with other engines, namely EA's propetiary Frostbite engine
@みやぞのさくら Its great feat that they managed to make such a small game engine, but too many games I've played from itch made in godot run like shit on my laptop. The performance of godot is so abysmally bad.
TH-camr: Let me know what you guys are interested in.
Me: **Screams in Gary Oldman** *EVERYTHINGGGGGGG*
me with 4 divorces: yeah I, too, am amazing at commitment and dedication
That's rough man.
That's rough buddy
deffinitely one of the best devs, I've ever seen
on TH-cam who's sharing knowledge to us for free.
Don't give up the engine, you can convert it in to a 3d modeling software, or something.
Wish you the best of luck. You are awesome!
"Making Game Engines is hard" You don't saaaaaaaaaaaay.
Thanks for reorganizing this channel back to c++/engine development.
So in terms of the Vulkan crash, when I'd get the device lost error, this was typically triggered in the destruct as the device is used to clean up resources, but the device was being used still wherever the crash occurred, so the ACTUAL problem is being masked. I'm curious which validation layers do you have enabled? All? Verbose? Also, have you tested this in RenderDoc? it spits out the validation layer errors as well, I've caught a few bugs that way as well.
Thanks for this video! I needed to hear that but at the end :)
I would love to see an in-depth video of how your shadow mapping works, like walk through the code in-depth
Another great video. Thx Cherno
This was such a cool video!! Please make more devlogs!!!
Thanks, for the encouragement. Keep working hard , hope we can see some games being produced in Hazel soon :P
Cherno, have you tried using NVidia NSight or RenderDoc to debug your GPU crash? I have used those before to debug and root cause DirectX crashes and would imagine they should help with Vulkan as well. If I were to guess, I imagine you might have an issue with data alignment, data packing, or accessing out-of-bounds memory in your Compute Shaders. Since also Vulkan gives you control over synchronization, I would double check your memory barriers as well (a buffer may contain incomplete data). Hope you figure it out. Cheers.
+1 for Nvidia Nsight. Amazing tool.
I think it was great and always looking forward to some technical solutions and examples :)
if You're trying to write abstract rendering api with differend backend look at bgfx library and api thats they use for abstracting rendering
Or GFX-HAL
I have no idea what you are saying, but I enjoy watching your contributions to the gaming industry
Somehow related, as a totally beginner I wanted to start making my own game engine with DirectX because I didn't like OpenGL that much so I decided to pick Frank Luna's book about DX12
In the beginning I was understanding what was going on but now that I've reached the second half of the book, I'm starting to feel very confused and tired of creating a lot of things, separating them only to implement one little thing.
Now I feel you when you dislike low level graphics programming, especially when I see some cool effects that people create in few minutes/hours in Unity or Unreal
But I still like DirectX and I still don't want to give up
If someone is an DirectX expert and/or has read Frank Luna's books, please write me back
Don't give up!! you can do it!!
@@lazergenix I have a lot of doubts and questions, so I'd like to talk about them with you.
Thanks
@@lazergenix where can i contact you?
@@ilieschamkar6767 ?
Yes, I'm interested in the under hood architecture/design stuff :)
You are the reason the word “dedication” was invented.
🥺😭
Oh nice! Guest Episode with Ed Sheeran!
How can I watch everything like step by step is it available in patreon or twitch?
it is available in patreon
@@justasydefix6251 thx for the reply
A error that occures randomly sounds like a race condition to me. You should probably check stuff like pipeline barriers and in general resource related things. Since the problem seems to be specific to the environment map code there may be something that is unique to that part of code. Sorry for any spelling mistakes :-/
That’s what I was thinking as well
I've been going about making engines for a few years now, it went from a quad to now an ecs. With that bit of experience I can say yes, game engines are really hard to make.
I am a Game Engine programmer myself, but I don't know a lot about rendering (I focus mainly on gameplay programming and tools). It would be cool if you made a video specifically about the various rendering APIs and what are the common points that could be exploted to write an API-indepentend renderer. I know a bit about oldschool OpenGL and some variants like EGL and NitroGL (from my good old PS3 and Nintendo DS days) but I don't know many of the most recent technologies and I could use some videos to wrap my head around those! Thanks and awesome video as always ^^
Have you heard about BGFX? I saw lots of games use it, like Minecraft: Bedrock Edition. It might get out of learning specific APIs but It really makes our lives more enjoyable.
Also there is GFX-HAL, It is good that we have open source low level cross platform rendering APIs, Personally I am currently learning Apple’s Metal API which is a nice API for learning low level rendering but when it comes to making a product I am definitely gonna use one of those cross platform APIs as I can definitely not learn all APIs for each platform and if it is good then why use specific API for each platform if we already have a good HAL (Hardware Abstraction Layer) rendering API
Rule #1.... it's always the changes that will effect nothing that will have subtle side-effects that will take weeks to track down. (As I have found out many times) Managed to get animated, instanced meshes working in mine now with decent lighting. ImGui was a turning point as being able to change stuff on-the-fly and having a nice interface for examining all loaded assets led to a lot of huge performance tweaks.
Got loading all my models from like 4-5 seconds in debug mode to almost instant mainly by using a fast base64 decoder on the vertex arrays. I was debating a full binary file format but they can be hard to debug and look at with a text editor. Most of the file is human readable except for blobs of base64 on a single line for the mesh data.
I'm always a bit skeptical of these thicker api wrappers for most anything where it would have any meaningful benefit. The reason being: for where it is similar it probably isn't too complicated, and where it is different the specialized apis are probably much more suited syntactically to the task of expressing what they are, something the generalizing thick wrapper must accomplish in a balancing act between the apis it abstracts. This obfuscates the important bits and trivializes the easy parts even further. I don't mind me some magical code here and there, but there is such a thing as too much magic.
About the Vulkan crash, I don't know if you've tried enabling GPU-Assisted Validation or Synchronization Validation? Those can be enabled with vkconfig or in source code. I've found many bugs by enabling them.
Do not give up! u dose awesome JOB!
I know you've only got Nvidia hardware, but it might be helpful to run the open source RADV driver for an AMD card on Linux and see whether it also crashes. There are things like shader trap handlers which may give you an idea of the issue, but even if you can't it's a high quality driver which you can ask for help with. If you can't get it building for Linux you could probably just use Wine.
Unfortunately I've lent my desktop with a RX580 to a friend, and only have my Nvidia work laptop with me now.
Another thing 1: Have you played around with validation layers?
Another thing 2: vkTrace/gfxreconstruct. That way you could send a trace of the commands to people to replay and potentially find the issue without having to download and compile your project themselves.
Really impressive - i like the idea of Abstraction. An API for all Apis :D
What's the name of the sky model he says at 15:50? Couldn't quite get it
I can very much relate to giving critical bugs some space and working on something that doesn't make you hate your life for a bit. I hadnt even realised it till you mentioned it; i'm currently writing my dissertation and that experience definitely going in there somewhere.
I have to admit I've liked your videos, since I'm making my own engine in C and OpenGL. I hit somewhat of a brick wall about cascading shadow maps, since you mentioned doing a video about that in more detail about how to do such a thing, so I will put my vote in to that if possible. ;) I have a large environment (It's a space combat game on planetary surfaces) so I'm looking for how to do a large area, and yet still have it look good when the player flies close to the surface and would see their own shadow or that of buildings they are going past. I also have dynamic lighting from the weapons and explosions, etc, so I'm not sure it's practical to add shadows to those light sources, but I would be happy with just the sun doing it. In anycase, I know you have your own series of things you need to do, and my problems aren't yours. But you wanted to know what people wanted to see, so I mentioned my vote. Thanks again for doing these videos!
Your Game Engine is so awesome, keep going...
5:20 So you mean like Qt's RHI but for game engines?
It's actually kinda cathartic to know that skilled professional devs also get bogged down and frustrated by bugs like that too. And I totally identify with the urge to just work on something else for a while. I think for me it's primarily that I need to work on something that will distract me and give me a quick path back to the feeling of winning, competence, and actually accomplishing something after hours/days/weeks of banging my head against something that I just can't seem to crack. Once I've built back up some confidence and motivation, I can eventually dig back into the frustrating stuff again. I recently had one that had me feeling like I was on the verge of just throwing up my hands and quitting programming altogether. After weeks of frustration I finally took a break from it and worked on new shiny features for a few days. Then when I attacked it again, I finally stumbled across the root cause. It was a gstreamer bug specific to its WASAPI backend that was breaking drag and drop functionality on Windows in Qt and causing crashes because it was randomly deleting OLE objects that didn't belong to it. Not exactly the place I'd go looking for a bug that looked like a UI issue. heh
I also want to program a game engine which can provide an immense cgi quality graphics in conventional system like ps4 xb1 and entry level gpus just like cryengine and rockstar rage engine ithink graphics programming is a emerging environment in software engineering I think its more great and challenging than basic software development through this we can actually understand a true concepts of physics
Driver crashing is mostly caused by bad sync and UB and those are the reasons I switched to messing with Vulkan on Rust instead of c++
Why would rust help with vulkan sync stuff?
I've just wanted to ask for more videos teaching graphics programming. There is OpenGL series, Vulkan and directX should be covered too. There is lack of this sort of content on youtube. We want to better understand graphics and it would be great to learn from You Cherno.
but talking about the hair is why i watched the video
Sometimes the best way to solve a problem is to walk away from it and come back to it with a fresh perspective.
We are not gonna talk about the hair !
I really recommend you to integrate Nvidia Aftermath SDK into your engine, because otherwise you're blind to crash reasons. Aftermath is a must for page faults and other hard crash debugging.
Hi dude! Usually when I get those Vulkan crash on my side is usually related to wrong memory access (uninitialized or not mapped texture areas or shaders trying to use UVs in an unitialized area memory)
And yes... Unluckily even wonderful tools like RenderDoc are not yet helpful there when crashing with those memory access issues
Yeah just do something else to avoid burnout!
Something simple and creative is the best way.
I got stuck on a project because I was so fixated on solving a bug without considering taking a break to clear my head.
And that led to a terrible burnout situation emotionally and physically which in turn made the project took much longer time than it supposed to.
Lol, I earned my lesson the hard!
Always take a break and start something entirely new, simple and fun, to help the brain get excited again.
Renew your mind as often as you can!
inspiring dedication
Can you do a podcast on your opinion on rust-lang? Very curious
When i submit my command buffer i got the same problem with Vulkan: VK_DEVICE_LOST, so i can't even render anything :(
shadow rendering through the plane does this happen on all objects? as that could be a potential performance killer.
looking good so far tho.
Give this man a break, he is making a GODAMN game engine
do not quit. I warn you. don't quit. this is a threat from a game dev who believes you have got some major talent even if it is a group project, this is too amazing to die. its almost looking like a unity clone. this is a good thing as unity works very well. fuck ue4 it sucks. I would love to start using you're engine when dx12 has support. please do work on all major versions of direct x 9+ ect as this will give it some major play in the game dev community. I cannot express how glad I am that you have this project going. more work = more likely your gonna get attention from other devs who would love this.
Bro can you give download link of hazel
I guess you are using the Vulkan validation layers?
I’m curious, does Hazel-Vulkan support the newest vulkan features?(Raytracing, MeshShaders) and compute shaders?
I want to know what your plan is for VR integration. It's super important to consider that at the beginning of engine creation
is that a new hair renderer you implemented? the shading and movement looks great, the red pops!
Did you dye your hair, or are you using a new LUT?
Hey, I wanted to know, when r u planning to make a tutorial on this aspect, I would like to know how u r able to choose backend at runtime
Since almost all features like textures, window, framebuffers etc are specific to the backend
5:25 there is no problem in computer science that can't be solved with another layer of abstraction
Hey Cherno, what did you use to make your engines GUI. Did you use something like Qt?
Good luck bro 🙏🙏
Maybe I could be of help with your vulkan crash.
The debugging ability is a real issue with vulkan unfortunately even with all the validation layers and the driver logs.
But after years of writing my own engine and doing very weird stuff using advanced vulkan API calls to the extreme, we end up with a brain that can have a very good understanding of what’s happening under the hood in the driver and what makes it crash.
Also, it is possible that your current crash is a driver bug (I have found quite a few in the past two years especially on the Ray Tracing features and I got them to fix it)
But if you’re using old Vulkan 1.1 very standard features and you are using latest validation layers, it’s most likely a bug in your code, or in the way your engine runs.
When going from OpenGL to Vulkan, an aspect that can bite you easily is memory management and more precisely deleting a buffer or image that is currently in use by a command buffer that has not yet finished, this WILL cause a DEVICE_LOST error.
This may seem quite obvious but we never know maybe you didn’t think of it and it’s what’s happening to your engine: This kind of bug is often caused by the fact that you may be running with two frames in flight at the same time so you should delete objects at least two frames AFTER the last frame that used it.
Let me know if this helps, or you may contact me and maybe I can try to help. I have never seen the source code of your engine tho.
Quick leave a mark here before something happens! Anyways i know this vid will be great!
Debugging gnarly bugs is fun, if you can just view it as part of the work rather than something blocking your work.
What was the name of the dynamic sky system you mentioned? Is it worth a video on? :)
The game X4 had a lot of issues with that error in combination with all 3000 series GPUs. There was a hotfix from Nvidia ( GeForce Hotfix Driver 461.33 (Released 1/19/21)), since then it runs fine for me.
I fully understand that such issues can cost a tremendous amount of time and nerves, but we all know, the next time you will eventually will solve that issue - either by finding the cause or by changing the way how cubemaps are loaded. So keep going and do more creative things, when bugs stops you. I do this in my personal projects all time and most of the time, it helps me getting a different perspective.
Speaking of personal projects, i am in the process of writing a multi renderer myself too, which at the moment is at OpenGL 2.1 level and i have no experience with vulkan. So i would be really interested how to abstract the video apis in the simplest way possible.
In my current project which simulates fluid particles i have very simple rendering, except the fluid rendering technique which uses SSF (Screen Space Fluid Rendering). But compared to any render technique you have written in hazel, SSF is childs play :D
Yea something similar happend to me, I made a 3d interactive walk with compute shaders effect and they run differently on my new GPU ( less particless ) than they were on my old GPU.
So I got new GPU and it looked worse. Now I am avoiding compute shaders xD grrrr. Hope I will fix that bug someday xD findout what going on.
How have you approached testing? What's the integration/unit test setup?