For anyone who doesn’t know, in the older references, core meant memory, so when the core is dumped is actually writing the current state of memory into a file
If I remember correctly, the term goes back to the 1950's when they used magnetic iron cores for memory. I'm sure I read it in a 'history of Unix' book of some sort.
@@PeterJepson123 read the same thing in "Unix - a History and a memoir" by Brian Kernighan. Recommend it to anyone interested in Computer and Software history
Print debugging still is the most universal way of debugging. You can do it local, you can do it remote you can do it in embedded, you can do it in a high volume processing code that would be just time consuming to step through (or when you don't know exactly where the problem is and can't put conditional breakpoints). Core files can only help with memory problems (when your code crashes), but won't help when it doesn't crash, but just produces wrong resutls.
Prints are great at looking at long sequences of events too, which in a debugger can be quite difficult to setup several break-points and then have to poke around the entire local variable state at every break. Prints get straight to the point.
@@TheGameMakeGuy Not just logfiles, quite often there's also options to place the software or a component into debug mode and have even more verbouse print debug outputs. In addition to also being able to generate a memory dump, a configuration dump and whatnot. Personally I think debugging should be proportional to the issue or risk at hand. I wouldn't read a core dump to find a misplaced closing bracket, but if a prod sever is crashing mysteriously when it's running on client hardware alongside other applications which naturally have not been perfectly tested for compatibility from development but aren't expected to cause issues then obviously a core dump becomes a highly appropriate troubleshooting option.
@@SerBallisterIt does matter how you use them, though. You (the programmer) have to actually understand what the code should be doing so you can add only necessary print statements that let you follow the process and see where it went wrong.
Core files and debuggers are indeed very useful. But honestly... the occasional debugging printf() is perfectly fine too. I've caught a lot of bugs that way without having to fire up a debugger.
Depending on the situation printf or equivalent might be only way without changing changing threading/timing. Running a full debugger isn't a light weight process.
Debugging complex systems with time dependent interactions (user input, network connections, video game AI, user interfaces etc) are pretty much impossible with a debugger. The biggest problem here is that the debugger is extremely invasive in terms of time. You hit a break point and it literally freezes your entire application. This is a bit of an issue if you have code that interacts with real time things, which lets face it, is most of the difficult work in software engineering. I do use debuggers where I can and where it's the best tool for the matter at hand. But I am here objecting to the click-bait. Yes, please carry on using logging for debugging, as most debugging of complex systems really requires it.
or you could actually watch the video and realize that it's about post-mortem debugging of core dumps and not time invasive interactive debugging with breakpoints.
@sverkeren Or you could realize that debugging tools or core dumps aren't the Swiss army knives of debugging and that a simple print statement may be the best option at the moment. Also, this video is advice for programming in C and may not be applicable to any programming language. Still fitting to call it clickbait
@@everydaynova663 really, both techniques should be in your toolbelt. cores can tell you things that print statements can not, and print statements can tell you things dumps can not. dumps tell you where you are, prints tell you how you got there.
Don't do "typedef unsigned int uint32_t". The size of an int is compiler dependant (although it usually is 32 bits). Include stdint.h if you *need* exact size integer types.
This man is the perfect programming TH-camr for me. I’ve noticed lately at my uni in my courses that all the other students complain at an assignment/task that needs to be done in C (we’re in Operating Systems and we just wrote the first part of a shell where we had to create and implement the cd, pwd, and exit commands, now we’re going to exec the other commands as well as their arguments, and the whole class whined out loud when they were told that was the next assignment due next week. He also teased at the possibility of making us write our own memory allocator, which got even more audible rejection from the other students). However, I get excited at the thought of it. I love writing in C, I love the challenges it brings. My professor even pointed it out to me when I went to see him during his office hour to discuss getting an internship, how to go about it, what to prepare for, etc. that I seemed to really enjoy working at a low level close to the hardware. So, as a guy who seemingly loves writing low-level code, finding a programming TH-camr whose channel name is literally “Low Level Learning” is the best thing that I’ve found on TH-cam this year so far.
After writing software for 40 years in many languages, from the highest to lowest levels, including patches to some of the software used in this video, here's what I've learned about debugging: 1. The best debugging technique is to just read the code. If the code is too complex to trace in your head, it probably needs to be rewritten anyway. 2. After that, nothing beats print style debugging. It's easy, simple, and works in almost any context. 3. Debuggers and profilers are nice luxuries to have sometimes, but are frequently not feasible.
6:04 I think I noticed a small error in the if statement the last index of an array is always the size of the array minus one because arrays starts at 0 (except in LUA). So the maximum index of the array is 99, and not 100. if you select a box with 100 as ID, your program will segfault because index 100 is out of bounds. To fix that, you have to either replace the ">" sign to ">=" sign, or you can do if (i > 99) { ... }.
Typical “off-by-one” error. Program perhaps won't segfault: static g_Boxes takes 6800 bytes, hence it'll span two 4 KiB pages with 1392 writable bytes left after it (speaking x86'ish).
For those unfamiliar with C: "@System64 MC" is correct - and the index should be checked against 99 or less. The index variable "i", declared as an "unsigned int" will not go below zero, so you don't have to worry about that. I could not see what the read_int() function does if the user inputs a negative number.
minor correction: "ulimit -c unlimited" means allow a core file be created with unlimited size, not "the kernel is allowed to produce an unlimited amount of core files"
For Linux C++ users, I really recommend compiling your project (-g -O0) against the backward-cpp library. You will have a nicely printed stack trace from your segfault. It is probably sufficient to understand where the bug comes from, else yes, use gdb and the core dump.
cross platform stack traces were meant to be in C++20, but alas, the committee moves too slow. There is source_location, which you can use for debugging. source_location also doubles as a scrappy way to achieve compile-time reflection! You can extract class member's names as strings by abusing auto and source_location.
@@DefaultBoyBand -Og enables function inlining, so not exactly the best thing for stack traces, you need to additionally specify -fno-inline alongside it
It would also be helpful to demonstrate how to write test cases and how best to separate into multiple modules. Also, you should check the number only once when read in from the user as well as stop using magic numbers. Might also want to demonstrate using `gdb` even if your program isn't crashing.
@@windowsos-exestoppedworkin5391 iam just starting,Iam interested in low level programming stuff... high level languages and frameworks... They hide a lot of details.
@@avg_user-dd2yb if you are interested then you should check out FPGAs, essentially reconfigurable hardware that you can use to make essentially any digital circuit like processors, the “game”, “Turing Complete” is good for this too, you start off making basic logic gates and latches and stuff and end up making a computer and programming it in assembly. If you are only interested in assembly then “Shenzhen IO” is good too. The most complete way to understand it at a very low level is to know how the processor itself works and be mostly able to build one.
When I switched from python and C# to C, one of the problems I had was that there's no stack traces when something goes wrong. But this is actually even more useful than a stack trace. Amazing
The gdb gives you the stack traces. You could type backtrace or just bt from the core. You can also type where at a breakpoint. In our products we caught signals, then processed the exception - servers are NOT allowed to crash. There again in C++ there are lots of ways to better handle exceptions than in C.
@@stephenhookings1985 Yeah, I meant stack traces that print out automatically when there's an exception. I figured you can get that from gdb, but you have to know it's there.
It's a whole lot harder to use the core file when a released product crashes (no debug symbols). It is also much harder to use core files to track down crashes in multi-threaded programs, and impossible if the problem is caused by a deadlock elsewhere in the code.
Compiling with symbols only adds to the binary's size, but doesn't actually affect execution speed, as the runtime linker knows not to bind the debugging information into the process image, which is to say: binaries should always be built with debugging symbols embedded in them, they don't hurt runtime performance in any way, and the few milliseconds of loading time can very well save much misery later.
@@anujmchitale it is ALWAYS an option; you could compile with debugging information, then use (g)objdump to extract that information into a separate file, which you could then use if you need to debug the binary. You should never compile without generating debug symbols.
@@AnnatarTheMaia A classic reply without knowing the slightest about the system that the person might be using. Debugging goes beyond the "binary" when talking about microcontrollers. It's the system as a whole that needs to be debugged, which includes hardware responses. This objdump approach doesn't work when doing realtime debugging on the microcontroller itself. If a kernel isn't running on it.
@@anujmchitale as someone who has learned to program by banging the hardware directly in the constrained system with only 64 KB of bank switched memory, on a microprocessor with only three eight bit registers, I thank you kindly for insulting me, and I want you to know that I took your response very personally.
I believe there’s still a bug. You can enter 100 and it still crashes as the index is 0-99 😉I find it remarkable that you remember all the the commands. I always have to look em up. Then again I hardly debug, my code is usually error free 😅 I actually just wrote my first ever C program on the Amiga to just open a window and draw random computed pixels. And not being able to run the binary on my Mac (Aros HDF required UAE version 4 which is not released on Mac yet). So I upload my cross compiled binary and awaited the first crash… but it worked, first time! Usually that’s a no go. Especially with a whole new windowing environment.
@@akaikangaroo Can you name ONE 😝🤪 And obviously I don’t agree because 0 makes sense because also memory starts at 0000 making it ONE is a silly western construct. Because we a westerners were too stupid to not define zero, unlike the Indian Al-Khowarizini did. For ever gatefuk to him, now programming zero index makes complete sense 😁
When I saw that "i > 100" code, I immediately thought it was wrong and came to write a comment, but of course someone has already pointed it out. It should have been "i >= 100".
You code on the Amiga, that's why your programs are usually error free, because the Amiga shapes and reinforces correct programming. That explains a lot. Someone who has never programmed on the Amiga has no chance in hell of ever understanding why that is so. Good on you!
Good video , i just saw a video from Dave Plummer (ex Microsoft software engineer) and i learned to use the f_s functions all the other F functions are not safe, can bus buffer overflow , the new way so use prtinf_s, the video of Dave is called Stupid C tricks , i saw it and he's right , has to do witn functions not taken in account buffer lengts, and not checking them , not adding null remintator and so on, there are new functions and they have in common that they are named as the original functions but added with _s , these are safe functions can not induce a buffer overflow, good explanation also from Dave.
For Windows fellows, the equivelant is a .dmp file that is either next to the program or some crashes folder, and Visual Studio would be used to open it
Scanf with the right precautions works fine. Sure fscanf or whatever it was is technically better security wise, but is slower and lacks features. Just make sure to specify how much scanf is supposed to take in.
Another way to avoid the print statements that get confusing is to use asserts. Forcibly fail when the conditions you're desiring aren't met, or the conditions you're seeing are. It's printf debugging but only one message prints, and it means it's much more comprehensible when something does go wrong. It doesn't always substitute a debugger, but they're good practice anyway, so it's a good idea to use them regardless of if you're debugging or not
Not always debugging used to fix crashes and sometimes gdb can't show proper backtraces due to absence of debug symbols or code version of the build. Also real-time tight programs will not let to stop themselves so prints is the only choice in many situations.
Something that print debugging allows, but no traditional debugger (to my knowledge) supports, is seeing how certain values in your program evolve over time, like in any kind of loop. Seeing the change over time allows me to figure out a potential pattern, and from there derive the issue way faster than most other traditional debugging techniques. This becomes even more valuable in a language like Rust which has really rich formatting support and prevents most types of crashes from the get go, so the majority of bugs I need to fix are logic bugs, where finding patterns is often pretty fast.
"...is seeing how certain values in your program evolve over time, like in any kind of loop." DTrace is what you want. You can trace the program live while it is running, with no sideeffects on your program. You can even build your own DTrace probe points into your program, and they won't even affect performance, because they won't trigger unless DTrace is used... And if you're on Windows, Microsoft has just added support for DTrace...
This is a great video thanks. I generally use ifdef debug printf() style to isolate an offending function and create a new entry point and test that function to destruction. The only debugging tool I've been using is valgrind --leak-check=full. This is great. Cheers mate.
@@LowLevelTV I actually learned a lot about low level programming in this channel (your videos give me the necessary motivation to research a lot more)
For this simple example, printing i would have found the bug sooner. However, I do see that this can be useful for truly complex bugs, but not necessarily better than print statements.
Holy shit, this is so useful! I cannot tell you the number of hours I’ve spent tracking down sigsegvs! I didn’t even know the (core dumped) meant anything 😂
@@Brad_Script Actually it's a term for a specific type of memory that was in use at some time in the distant past, magnetic-core memory, usually shortened to core memory, where data was stored on magnetic rings.
Best tool for me yet: Dtrace. Once ported a large library from Linux and could remove 30% of all lines (trace/debug code). Replaced it with some Dtrace scripts, to have equivalent functionality, no need to recompile with instrumentation.
Or you can gcc -fsanitize=address to see where the segmentation dump core occurred, we use this method alot to understand where the pointer has failed. If not this, valgrind is the second choice of command to understand memory leaks.
This is great if you’re writing for a PC environment. Can’t use core files on embedded, a debugger and a robust logging system is the best to debug quickly. Also 99.99% opening up the assembly to see what’s going on is just to take more time than looking at the source
That's correct. It was completely unnecessary for this example and probably is almost never necessary as the compiler usually knows pretty well what it does with how it uses the registers. If you ever happen to go that deep you most likely want to turn around and check your bug on a higher level.
Agreed. While useful for specific problems, it’s a very security researcher mindset which is not useful for the debugging needs for the vast amount of software devs out there, especially on nontrivial projects unlike this toy example.
I was quite proficient with the debuggers I used most frequently, but I still use printf debugging for several reasons: 1. When there's no debugger available. 2. When the only available debuggers are ones I don't know how to use. 3. When the bug shows up only at high levels of optimization, where the connection between the code I wrote and the behavior of the resulting program as shown by the debugger becomes incomprehensible. In particular, if I need to know the current value of a variable, the compiler often tells me that the variable has been optimized away. If I insert a debugging printf, that will either disable the optimization, or at least produces the same debugging output as if the variable had not been optimized away.
I've been patiently waiting till I finally stumbled across someone who explains these concepts as well as you. Thank you so much sir! I am so excited to learn with you :)
Cool trick - I think I used this once a long time ago. Typically I just hit run on my IDE and it stops the debugger automatically on the line where the crash happens. Takes about 5 seconds to fix a bug like this.
Wow, this was a really well presented video. I never used this feature before, because I thought stuff like that would be overkill in most scenarios. But the way you showcased it made it seem pretty handy, while note being as complicated as I imagined. I will definitely try this out next time I code something.
Although I agree with the concepts in relation to crashing programs ...the use of printf or console debugging is still the best answer to figure out complex bugs like drag and drop and other scenarios where you do NOT want to stop execution but observe the flow. Also the best thing I saw done for a memory corruption problem was s senior engineer wrote a memory manager that would allocate restricted blocks around every block requested...so while running BAM it would trigger the debugger right when the memory was stepped on.
Starting in debug mode often takes 2-3 times longer so a log is usually more efficient. I only reach for the debugger when logs fail or are too cumbersome.
FWIW. “ulimit -c unlimited” doesn’t relate to the number of core files, but to the maximum size of a core file. If the maximum size is unlimited, it means you’ll get a core dump regardless of the “core” (memory) size of the process. It usually defaults to zero in a new shell which essentially disables core dumps since all cores are larger than zero bytes.
I usually add a couple of debugging options to programs, even shell scripts, which enable either some basic logging or that plus an overkill data dump. That way I can trace behavior without code changes. A "debugger" isn't always viable for whatever reason.
I used printf debugging over the weekend, of course, it was in a M1 kext, so no core to get, and apple kext lldb is very poor on M1, and panic dump sending is broken. It's not always the wrong thing to do, if you have no other choice :) But when it works, it's a beautiful thing!
The funniest experience of learning c is starting using print statements for debugging. Learning about debuggers and perror. End up programming drivers going back to using printk to print statements in order to debug :p
so much to replicate something visual studio has out of the box lol. i didnt even realize fully that i kept running my VS program in debug mode all the time
Extra tip, if you have compiled with -g you should have debug symbols too and you can see the crashed code in gdb with "list". But why live like this? This is the 21st century and if you have a repro case like that you can debug it directly in VS code, and see all variables right there. Core files are nice when either the repro is difficult or you are debugging someone else's crash. The assembly part of the video was a bit unnecessary imo. You won't be able to figure out the array size easily by looking at the registers.
Yeah, this is my primary complaint about programming in Linux/GCC: everything is just harder then MSVC. Heck, when you *have* to support Linux nowadays, I just code in MSVC on Windows and use CLANG to do the compilation on Windows, then once that works just use CLANG on Linux to generate the executable. I can't imagine having to actually DEVELOP on Linux using either GCC or CLANG.
Ive spent 30 years working in software development and never had an issue with print debugging. In fact its got me out of many holes. And sometimes its the only option if running the code on the server is the only option and you aren't able to run a debugger or similar on the server.
You don't need that 1. Print will always show the error before after point 2. Call all your function parameters with non user input 3. Make sure to store all user input, database reads or API calls in unlimited memory(or limit memory and handle it gracefully) then pass it to your functions 4. Call all your functions in a debug caller 5. Print your own core file using the steps above to debug Now he used uint32 which is more than you can type in a minute. But if you cat a few GB of number to input it will exceed uint32
Every function call in a debug caller will log it's parameters to a log file. Don't make it with an extension .log just dump it to an extension lead file
Core files have their place. However, printf has one key advantage IMO over any debugger: repeatability. When you use a debugger you click or give commands and that is a one time operation. If you can't figure out the problem you need to run the code again and repeat all that debugging. Where as whatever you put in printf will be there on every run until you remove them. I find that invaluable. I do use a debugger but most of the time for the kind of work I do printf & friends are much more useful. They are kind of scriptable debugger in a language you know and love.
@@fr5229 I was referring to debugger in general, not core files. Have never used core files so I could not really comment on them. However I would presume that they are a (massive) snap shot at given time(s) where as a carefully placed print statement IMO are much less work. But each to their own poisson. I use a debugger but many many times I find print statements much more useful.
@@Axel_Andersen Oh I see. Yeah you’re right- core files are very cumbersome, overkill, the last resort. I avoid needing to use them and I’m actually surprised that this video is calling for people to be so impractical just because they can. I’m not really opinionated either way about print statements vs debuggers, but you should check whether “time travel debuggers” are available to you
@@fr5229 The true benefit of printing over core files/debuggers is that printing is actually usable in production. Virtually every non-trivial piece of software has a logger, even in release builds. Virtually none of them compile with debug flags. Core dumps and the debugger are essentially useless in optimized binaries, which is most software.
Well, "the box" was obviously a reference to the movie 7, and the "Dave" was a reference to the "2001: a space odyssey". Is there something like that for php as well, or do the test units have to do everything?
I would say that using `-fsanitize=address,undefined` as a flag when compiling the executable will work just as well or better the vast majority of the time (why try to recover the state of the program from partial information after the fact when you can just insert a bunch of validation to catch what would go wrong before it does?) with much less hassle. Core file debugging lets you analyze segmentation faults, sure, but that's about all it does, and - address sanitizer does it arguably better - print debugging lets you do other things too
For anyone who doesn’t know, in the older references, core meant memory, so when the core is dumped is actually writing the current state of memory into a file
👍
If I remember correctly, the term goes back to the 1950's when they used magnetic iron cores for memory. I'm sure I read it in a 'history of Unix' book of some sort.
@@PeterJepson123 read the same thing in "Unix - a History and a memoir" by Brian Kernighan. Recommend it to anyone interested in Computer and Software history
That is true! Lovely book all around. Kernighan is such an amazing person!
@@PeterJepson123 My mom worked on those back in her college days.
Print debugging still is the most universal way of debugging. You can do it local, you can do it remote you can do it in embedded, you can do it in a high volume processing code that would be just time consuming to step through (or when you don't know exactly where the problem is and can't put conditional breakpoints). Core files can only help with memory problems (when your code crashes), but won't help when it doesn't crash, but just produces wrong resutls.
Prints are great at looking at long sequences of events too, which in a debugger can be quite difficult to setup several break-points and then have to poke around the entire local variable state at every break. Prints get straight to the point.
@@TheGameMakeGuy Not just logfiles, quite often there's also options to place the software or a component into debug mode and have even more verbouse print debug outputs. In addition to also being able to generate a memory dump, a configuration dump and whatnot. Personally I think debugging should be proportional to the issue or risk at hand. I wouldn't read a core dump to find a misplaced closing bracket, but if a prod sever is crashing mysteriously when it's running on client hardware alongside other applications which naturally have not been perfectly tested for compatibility from development but aren't expected to cause issues then obviously a core dump becomes a highly appropriate troubleshooting option.
You can also use /assertions/ if the language has that feature.
@@SerBallisterIt does matter how you use them, though.
You (the programmer) have to actually understand what the code should be doing so you can add only necessary print statements that let you follow the process and see where it went wrong.
Yeah this core debugging is useful but doesn’t replace other forms of debugging.
6:13 Fun fact, this is a mistake. As this array's length is 100, its maximum value is 99, so entering 100 will cause a crash anyway.
I would have used >= 100
@@JosefdeJoanelli Yes, but he didn't do it.
@@glowiak3430I know
Nice! I came to comments to see if anyone else caught this off-by-1 error
I assume it was intentional to drive engagement, but yet here I am.
Core files and debuggers are indeed very useful. But honestly... the occasional debugging printf() is perfectly fine too. I've caught a lot of bugs that way without having to fire up a debugger.
Depending on the situation printf or equivalent might be only way without changing changing threading/timing. Running a full debugger isn't a light weight process.
Sometimes, you just can't use gdb, like when trying to debug bare metal Raspberry Pi code, so you have to do print debugging over UART.
Or LED blinking :) while the best gdb for bare metal is the oscilloscope :D Instead of printf I know preferring the dbg-macro library for C++
@@pikachuchujelly4119 I'm using gdb to debug stm32, don't know what are you talking about
@@pikachuchujelly4119 That might be the easiest option, but it's most assuredly not the only way. You could probably use the GPIO and LEDs too.
Debugging complex systems with time dependent interactions (user input, network connections, video game AI, user interfaces etc) are pretty much impossible with a debugger. The biggest problem here is that the debugger is extremely invasive in terms of time. You hit a break point and it literally freezes your entire application. This is a bit of an issue if you have code that interacts with real time things, which lets face it, is most of the difficult work in software engineering.
I do use debuggers where I can and where it's the best tool for the matter at hand. But I am here objecting to the click-bait. Yes, please carry on using logging for debugging, as most debugging of complex systems really requires it.
or you could actually watch the video and realize that it's about post-mortem debugging of core dumps and not time invasive interactive debugging with breakpoints.
@sverkeren Or you could realize that debugging tools or core dumps aren't the Swiss army knives of debugging and that a simple print statement may be the best option at the moment. Also, this video is advice for programming in C and may not be applicable to any programming language. Still fitting to call it clickbait
@@everydaynova663 really, both techniques should be in your toolbelt. cores can tell you things that print statements can not, and print statements can tell you things dumps can not. dumps tell you where you are, prints tell you how you got there.
Absolutely right !
@@sverkeren ah, but I did not watch the video because I do not like clickbait. See...
Don't do "typedef unsigned int uint32_t". The size of an int is compiler dependant (although it usually is 32 bits). Include stdint.h if you *need* exact size integer types.
Yeah, stdint.h has been around for _how_ long now? Even VS 6 had some homebrew versions made for it at some point.
or just program in Rust
@@dmitryhetman1509 ikr, Go ftw
@@biigsmokee if you've moved to Rust because of fixed width integer types you're probably not staying.
plus it makes your program more portable technically
This man is the perfect programming TH-camr for me. I’ve noticed lately at my uni in my courses that all the other students complain at an assignment/task that needs to be done in C (we’re in Operating Systems and we just wrote the first part of a shell where we had to create and implement the cd, pwd, and exit commands, now we’re going to exec the other commands as well as their arguments, and the whole class whined out loud when they were told that was the next assignment due next week. He also teased at the possibility of making us write our own memory allocator, which got even more audible rejection from the other students). However, I get excited at the thought of it. I love writing in C, I love the challenges it brings. My professor even pointed it out to me when I went to see him during his office hour to discuss getting an internship, how to go about it, what to prepare for, etc. that I seemed to really enjoy working at a low level close to the hardware. So, as a guy who seemingly loves writing low-level code, finding a programming TH-camr whose channel name is literally “Low Level Learning” is the best thing that I’ve found on TH-cam this year so far.
After writing software for 40 years in many languages, from the highest to lowest levels, including patches to some of the software used in this video, here's what I've learned about debugging:
1. The best debugging technique is to just read the code. If the code is too complex to trace in your head, it probably needs to be rewritten anyway.
2. After that, nothing beats print style debugging. It's easy, simple, and works in almost any context.
3. Debuggers and profilers are nice luxuries to have sometimes, but are frequently not feasible.
So you're saying a nice pretty-printer is more important than a nice debugger? 👀
This is very basic debugging for crashes but the video title sounded like more. Print statements to deal with non fatal errors are still quite useful.
100% agree. This is probably the first comment I've seen from you where you don't sound angry lol
@@vastabyss6496 I'm not angry generally. Just the way things come across via text on the web probably makes it seem that way.
6:04 I think I noticed a small error in the if statement
the last index of an array is always the size of the array minus one because arrays starts at 0 (except in LUA). So the maximum index of the array is 99, and not 100. if you select a box with 100 as ID, your program will segfault because index 100 is out of bounds. To fix that, you have to either replace the ">" sign to ">=" sign, or you can do if (i > 99) { ... }.
Typical “off-by-one” error. Program perhaps won't segfault: static g_Boxes takes 6800 bytes, hence it'll span two 4 KiB pages with 1392 writable bytes left after it (speaking x86'ish).
@@-wx-78- Yeah, but it might overwrite unrelated memory.
@@chlorobyte_projects Definitely.
Indexes in LUA start at 1? Absolute madness.
For those unfamiliar with C: "@System64 MC" is correct - and the index should be checked against 99 or less. The index variable "i", declared as an "unsigned int" will not go below zero, so you don't have to worry about that. I could not see what the read_int() function does if the user inputs a negative number.
Already using gdb on low level stuff, it’s ironic… thanks a lot man I’m really blessed by ur channel.
minor correction: "ulimit -c unlimited" means allow a core file be created with unlimited size, not "the kernel is allowed to produce an unlimited amount of core files"
For Linux C++ users, I really recommend compiling your project (-g -O0) against the backward-cpp library. You will have a nicely printed stack trace from your segfault. It is probably sufficient to understand where the bug comes from, else yes, use gdb and the core dump.
cross platform stack traces were meant to be in C++20, but alas, the committee moves too slow. There is source_location, which you can use for debugging. source_location also doubles as a scrappy way to achieve compile-time reflection! You can extract class member's names as strings by abusing auto and source_location.
use -fsanitize=address
-Og is better for debugging (just a tip)
@@DefaultBoyBand -Og enables function inlining, so not exactly the best thing for stack traces, you need to additionally specify -fno-inline alongside it
@@atijohn8135 huh... didn't know that! thanks
It would also be helpful to demonstrate how to write test cases and how best to separate into multiple modules. Also, you should check the number only once when read in from the user as well as stop using magic numbers. Might also want to demonstrate using `gdb` even if your program isn't crashing.
0:12 no, this has never been me, because, I have (honestly) never been in a situation where I couldn’t find the problem using print statements
This is super helpful for beginners as well! Thanks a lot!!
You're so welcome!
@@LowLevelTV can you make a tutorial on making basic interpreter and compilers using c,would be really helpful.
I absolutely agree, I keep trying to write a compiler in C but I suck so I eventually give up and then I start over and the cycle repeats
@@windowsos-exestoppedworkin5391 iam just starting,Iam interested in low level programming stuff... high level languages and frameworks... They hide a lot of details.
@@avg_user-dd2yb if you are interested then you should check out FPGAs, essentially reconfigurable hardware that you can use to make essentially any digital circuit like processors, the “game”, “Turing Complete” is good for this too, you start off making basic logic gates and latches and stuff and end up making a computer and programming it in assembly. If you are only interested in assembly then “Shenzhen IO” is good too.
The most complete way to understand it at a very low level is to know how the processor itself works and be mostly able to build one.
When I switched from python and C# to C, one of the problems I had was that there's no stack traces when something goes wrong. But this is actually even more useful than a stack trace. Amazing
The gdb gives you the stack traces. You could type backtrace or just bt from the core.
You can also type where at a breakpoint.
In our products we caught signals, then processed the exception - servers are NOT allowed to crash. There again in C++ there are lots of ways to better handle exceptions than in C.
@@stephenhookings1985 Yeah, I meant stack traces that print out automatically when there's an exception. I figured you can get that from gdb, but you have to know it's there.
@@ishashkaor, you know how to find it. Manuals are amazing.
I like how you said we can look at the assembly instructions to debug this and resigned from doing so as soon as you saw them
It's a whole lot harder to use the core file when a released product crashes (no debug symbols). It is also much harder to use core files to track down crashes in multi-threaded programs, and impossible if the problem is caused by a deadlock elsewhere in the code.
And absolutely not an option for me, programming for microcontrollers. 😂
Compiling with symbols only adds to the binary's size, but doesn't actually affect execution speed, as the runtime linker knows not to bind the debugging information into the process image, which is to say: binaries should always be built with debugging symbols embedded in them, they don't hurt runtime performance in any way, and the few milliseconds of loading time can very well save much misery later.
@@anujmchitale it is ALWAYS an option; you could compile with debugging information, then use (g)objdump to extract that information into a separate file, which you could then use if you need to debug the binary. You should never compile without generating debug symbols.
@@AnnatarTheMaia A classic reply without knowing the slightest about the system that the person might be using.
Debugging goes beyond the "binary" when talking about microcontrollers.
It's the system as a whole that needs to be debugged, which includes hardware responses.
This objdump approach doesn't work when doing realtime debugging on the microcontroller itself. If a kernel isn't running on it.
@@anujmchitale as someone who has learned to program by banging the hardware directly in the constrained system with only 64 KB of bank switched memory, on a microprocessor with only three eight bit registers, I thank you kindly for insulting me, and I want you to know that I took your response very personally.
I believe there’s still a bug. You can enter 100 and it still crashes as the index is 0-99 😉I find it remarkable that you remember all the the commands. I always have to look em up. Then again I hardly debug, my code is usually error free 😅
I actually just wrote my first ever C program on the Amiga to just open a window and draw random computed pixels. And not being able to run the binary on my Mac (Aros HDF required UAE version 4 which is not released on Mac yet). So I upload my cross compiled binary and awaited the first crash… but it worked, first time! Usually that’s a no go. Especially with a whole new windowing environment.
That's why one-indexed languages rule🙂
@@akaikangaroo Can you name ONE 😝🤪 And obviously I don’t agree because 0 makes sense because also memory starts at 0000 making it ONE is a silly western construct. Because we a westerners were too stupid to not define zero, unlike the Indian Al-Khowarizini did. For ever gatefuk to him, now programming zero index makes complete sense 😁
@@CallousCoder Lua, Julia and 18 more😜
When I saw that "i > 100" code, I immediately thought it was wrong and came to write a comment, but of course someone has already pointed it out. It should have been "i >= 100".
You code on the Amiga, that's why your programs are usually error free, because the Amiga shapes and reinforces correct programming. That explains a lot. Someone who has never programmed on the Amiga has no chance in hell of ever understanding why that is so. Good on you!
if i knew this in my first year where i've learned trees and graphs in C imagine all the seg faults
thank you for this!
Good video , i just saw a video from Dave Plummer (ex Microsoft software engineer) and i learned to use the f_s functions all the other F functions are not safe, can bus buffer overflow , the new way so use prtinf_s, the video of Dave is called Stupid C tricks , i saw it and he's right , has to do witn functions not taken in account buffer lengts, and not checking them , not adding null remintator and so on, there are new functions and they have in common that they are named as the original functions but added with _s , these are safe functions can not induce a buffer overflow, good explanation also from Dave.
Woah this is cool! I wish I'd known about this when I still wrote in C back in uni. Great video!
Thank you!
For Windows fellows, the equivelant is a .dmp file that is either next to the program or some crashes folder, and Visual Studio would be used to open it
What's the Windows' equivalent of the runtime linker, ld.so.1?
Also, don't use scanf. Use the secure variants, and try to reject input as soon as it is parsed, not later on as per his example.
Scanf with the right precautions works fine. Sure fscanf or whatever it was is technically better security wise, but is slower and lacks features. Just make sure to specify how much scanf is supposed to take in.
What are the secure variants?
Another way to avoid the print statements that get confusing is to use asserts. Forcibly fail when the conditions you're desiring aren't met, or the conditions you're seeing are. It's printf debugging but only one message prints, and it means it's much more comprehensible when something does go wrong. It doesn't always substitute a debugger, but they're good practice anyway, so it's a good idea to use them regardless of if you're debugging or not
Not always debugging used to fix crashes and sometimes gdb can't show proper backtraces due to absence of debug symbols or code version of the build. Also real-time tight programs will not let to stop themselves so prints is the only choice in many situations.
TH-cam programmers: COME OUT WITH YOUR PRINT STATEMENTS UP
Me: You'll never take my print statements, copper! NEVER!
5:18 in gdb is used AT&T assembly notation ; INSTRUCTION ORIGIN, DESTINATION ; and not in INTEL (more used) assembly notation (aka. INST DEST, ORIG)
Something that print debugging allows, but no traditional debugger (to my knowledge) supports, is seeing how certain values in your program evolve over time, like in any kind of loop. Seeing the change over time allows me to figure out a potential pattern, and from there derive the issue way faster than most other traditional debugging techniques.
This becomes even more valuable in a language like Rust which has really rich formatting support and prevents most types of crashes from the get go, so the majority of bugs I need to fix are logic bugs, where finding patterns is often pretty fast.
You can have debuggers stop every time a value changes for a variable. I think that does exactly what you're talking about.
"...is seeing how certain values in your program evolve over time, like in any kind of loop." DTrace is what you want. You can trace the program live while it is running, with no sideeffects on your program. You can even build your own DTrace probe points into your program, and they won't even affect performance, because they won't trigger unless DTrace is used... And if you're on Windows, Microsoft has just added support for DTrace...
This is a great video thanks. I generally use ifdef debug printf() style to isolate an offending function and create a new entry point and test that function to destruction. The only debugging tool I've been using is valgrind --leak-check=full. This is great. Cheers mate.
Thanks for sharing!
Holy mother of valgrind, I was just struggling with debugging a program when you dropped this video...now my program works. Thanks :)
Glad it helped!
@@LowLevelTV I actually learned a lot about low level programming in this channel (your videos give me the necessary motivation to research a lot more)
Great video Dave! Informative as always.
Thanks for watching
2:01 yeah, hacking string with segment fault then u core dump and other things that i forgot
For this simple example, printing i would have found the bug sooner. However, I do see that this can be useful for truly complex bugs, but not necessarily better than print statements.
Holy shit, this is so useful! I cannot tell you the number of hours I’ve spent tracking down sigsegvs! I didn’t even know the (core dumped) meant anything 😂
5:21, the mov instruction actually does the opposite of what you said, because it's in AT&T syntax.
5:22 it's move [rdx + rax] *into* eax, not the other way around
I never knew what "core dumped" meant, this makes so much more sense now! Thank you :)
core is archaic term for memory
@@Brad_Script Actually it's a term for a specific type of memory that was in use at some time in the distant past, magnetic-core memory, usually shortened to core memory, where data was stored on magnetic rings.
Best tool for me yet: Dtrace.
Once ported a large library from Linux and could remove 30% of all lines (trace/debug code). Replaced it with some Dtrace scripts, to have equivalent functionality, no need to recompile with instrumentation.
You also can use the flag -fsanitize=address -g flag for these seg fault , buffer overflow, etc...
I wish I knew this 25 years ago. I had to dwbug the code differently. It would have greatly improved my debugging speed.
Or you can gcc -fsanitize=address to see where the segmentation dump core occurred, we use this method alot to understand where the pointer has failed. If not this, valgrind is the second choice of command to understand memory leaks.
Won't your program still crash for i = 100?
it will but it's just a small oversight on his part, it's not critical to the topic at hand
Nice syntax, I really like when someone takes care of readability and looks at the same time!
Excellent tutorial. Although not a solution for every bug, it's still a great way to debug C memory-related issues.
Half a semester of debugging seg faults, why did I not look this up earlier? THE PAIN!
recently i've been using assert alot more, and that's been much better than print statements for some usecases
Hint : use sudo -i instead of sudo su. Same experience except all commands you’ll use can be logged properly as a sudo use.
This is great if you’re writing for a PC environment. Can’t use core files on embedded, a debugger and a robust logging system is the best to debug quickly. Also 99.99% opening up the assembly to see what’s going on is just to take more time than looking at the source
I use coredumpctl on Linux. I didn't know you can change the directory where core dumps are saved. Very interesting. Thanks 🙏
What is the point of looking at the assembly instructions and register states? To me, it looked like you gained nothing from looking at them.
That's correct. It was completely unnecessary for this example and probably is almost never necessary as the compiler usually knows pretty well what it does with how it uses the registers. If you ever happen to go that deep you most likely want to turn around and check your bug on a higher level.
Agreed. While useful for specific problems, it’s a very security researcher mindset which is not useful for the debugging needs for the vast amount of software devs out there, especially on nontrivial projects unlike this toy example.
I was quite proficient with the debuggers I used most frequently, but I still use printf debugging for several reasons:
1. When there's no debugger available.
2. When the only available debuggers are ones I don't know how to use.
3. When the bug shows up only at high levels of optimization, where the connection between the code I wrote and the behavior of the resulting program as shown by the debugger becomes incomprehensible. In particular, if I need to know the current value of a variable, the compiler often tells me that the variable has been optimized away. If I insert a debugging printf, that will either disable the optimization, or at least produces the same debugging output as if the variable had not been optimized away.
You should at least fix it properly with >= 100… instead of > 100
Wanted to write the same thing
But the index starts at zero, 100 will overflow
printing variables is pretty useful without needing to understand registers or set breakpoints. i personally use both.
I've been patiently waiting till I finally stumbled across someone who explains these concepts as well as you. Thank you so much sir! I am so excited to learn with you :)
Cool trick - I think I used this once a long time ago.
Typically I just hit run on my IDE and it stops the debugger automatically on the line where the crash happens. Takes about 5 seconds to fix a bug like this.
Wow, this was a really well presented video. I never used this feature before, because I thought stuff like that would be overkill in most scenarios. But the way you showcased it made it seem pretty handy, while note being as complicated as I imagined. I will definitely try this out next time I code something.
Try gdb Save and reverse execution functionalities. It's pain to learn, but super usefull with the worst bugs.
Can't do this on most Microcontrollers, but I'll keep it in mind whenever I write a PC program in C.
I didn't know you can actually configure core like that. This is pretty useful.
Although I agree with the concepts in relation to crashing programs ...the use of printf or console debugging is still the best answer to figure out complex bugs like drag and drop and other scenarios where you do NOT want to stop execution but observe the flow.
Also the best thing I saw done for a memory corruption problem was s senior engineer wrote a memory manager that would allocate restricted blocks around every block requested...so while running BAM it would trigger the debugger right when the memory was stepped on.
Starting in debug mode often takes 2-3 times longer so a log is usually more efficient. I only reach for the debugger when logs fail or are too cumbersome.
FWIW. “ulimit -c unlimited” doesn’t relate to the number of core files, but to the maximum size of a core file. If the maximum size is unlimited, it means you’ll get a core dump regardless of the “core” (memory) size of the process. It usually defaults to zero in a new shell which essentially disables core dumps since all cores are larger than zero bytes.
Wow today i learned why some say tack instead of dash. Interesting. Still makes my eye twitch every time I hear it.
Beside using core dump files is a useful and really powerful debugging technique. Using some Logging or print statements stay still useful too.
03:16 Security hint:
Always use "su -" when changing to root to avoid keeping a compromised environment.
What I'm trying to understand here is how the values in rax and rdx gave away the nature of the issue.
Excuse me, but I add a bunch of prints to figure out what's wrong and eventually I DO crack the code. Works every time.
Plenty of things print statements can't do
In modern IDEs you can also set breakpoints that log something
I usually add a couple of debugging options to programs, even shell scripts, which enable either some basic logging or that plus an overkill data dump. That way I can trace behavior without code changes. A "debugger" isn't always viable for whatever reason.
Really nice ! This is why I want to learn assembly it's really helpful in all cases
Glad it was helpful!
I used printf debugging over the weekend, of course, it was in a M1 kext, so no core to get, and apple kext lldb is very poor on M1, and panic dump sending is broken. It's not always the wrong thing to do, if you have no other choice :) But when it works, it's a beautiful thing!
although gdb doesnt work on m1 macs afaik, lldb is as good tho
Yeah it's been years and still no gdb support on Apple silicon.
What's really fun is when your program cores with the -o option, but doesn't when you specify -g!
The funniest experience of learning c is starting using print statements for debugging. Learning about debuggers and perror. End up programming drivers going back to using printk to print statements in order to debug :p
The reason I do print debugging is rarely to debug crashes, it's to understand the flow of logic.
so much to replicate something visual studio has out of the box lol. i didnt even realize fully that i kept running my VS program in debug mode all the time
I'll stay loyal to my prints, thank you
Thank you so much... it's really hard to find good content about gdb.
Happy to help!
gdb is built-in in xcode runtime for mac users if any bug exist in runtime of their compiled c code
I will be using print debugging and you cant stop me. It is the best and only way to debug.
if(i >= 100){
printf("....");
return;
}
Not just greater.
Love ur videos!
Still allowing index 100 and that's bad too :P
Extra tip, if you have compiled with -g you should have debug symbols too and you can see the crashed code in gdb with "list".
But why live like this? This is the 21st century and if you have a repro case like that you can debug it directly in VS code, and see all variables right there. Core files are nice when either the repro is difficult or you are debugging someone else's crash.
The assembly part of the video was a bit unnecessary imo. You won't be able to figure out the array size easily by looking at the registers.
Yeah, this is my primary complaint about programming in Linux/GCC: everything is just harder then MSVC. Heck, when you *have* to support Linux nowadays, I just code in MSVC on Windows and use CLANG to do the compilation on Windows, then once that works just use CLANG on Linux to generate the executable. I can't imagine having to actually DEVELOP on Linux using either GCC or CLANG.
Nice explanation of debugging. Thanks!
This process is great for complied languages, not so easy to do for interpreted languages.
Ah, Seven reference.
ayyy this guy gets it
Took me a while to get it. Wow, thanks 😹😹😹
I’m about to start using GDB in my codebase - thank you so much 👍🏻
Ive spent 30 years working in software development and never had an issue with print debugging. In fact its got me out of many holes. And sometimes its the only option if running the code on the server is the only option and you aren't able to run a debugger or similar on the server.
An address sanitizer compile with -g would help lots as well
You don't need that
1. Print will always show the error before after point
2. Call all your function parameters with non user input
3. Make sure to store all user input, database reads or API calls in unlimited memory(or limit memory and handle it gracefully) then pass it to your functions
4. Call all your functions in a debug caller
5. Print your own core file using the steps above to debug
Now he used uint32 which is more than you can type in a minute. But if you cat a few GB of number to input it will exceed uint32
Every function call in a debug caller will log it's parameters to a log file. Don't make it with an extension .log just dump it to an extension lead file
* extension less
Core files have their place. However, printf has one key advantage IMO over any debugger: repeatability. When you use a debugger you click or give commands and that is a one time operation. If you can't figure out the problem you need to run the code again and repeat all that debugging. Where as whatever you put in printf will be there on every run until you remove them. I find that invaluable. I do use a debugger but most of the time for the kind of work I do printf & friends are much more useful. They are kind of scriptable debugger in a language you know and love.
If anything, core files give you repeatability whereas print statements don’t, unless your application is trivial/deterministic.
@@fr5229 I was referring to debugger in general, not core files. Have never used core files so I could not really comment on them. However I would presume that they are a (massive) snap shot at given time(s) where as a carefully placed print statement IMO are much less work. But each to their own poisson. I use a debugger but many many times I find print statements much more useful.
@@Axel_Andersen Oh I see.
Yeah you’re right- core files are very cumbersome, overkill, the last resort. I avoid needing to use them and I’m actually surprised that this video is calling for people to be so impractical just because they can.
I’m not really opinionated either way about print statements vs debuggers, but you should check whether “time travel debuggers” are available to you
@@fr5229 The true benefit of printing over core files/debuggers is that printing is actually usable in production. Virtually every non-trivial piece of software has a logger, even in release builds. Virtually none of them compile with debug flags. Core dumps and the debugger are essentially useless in optimized binaries, which is most software.
Really it goes to the core!
Printf is one of the most useful tools to debug, and you cannot tell me otherwise.
Well, "the box" was obviously a reference to the movie 7, and the "Dave" was a reference to the "2001: a space odyssey".
Is there something like that for php as well, or do the test units have to do everything?
DTrace.
I would say that using `-fsanitize=address,undefined` as a flag when compiling the executable will work just as well or better the vast majority of the time (why try to recover the state of the program from partial information after the fact when you can just insert a bunch of validation to catch what would go wrong before it does?) with much less hassle. Core file debugging lets you analyze segmentation faults, sure, but that's about all it does, and
- address sanitizer does it arguably better
- print debugging lets you do other things too
„we can actually look at assembly instructions to figure out what went wrong“
Proceeds to briefly show assembly but not actually use it
As a computer engineering student and with knowing the assembly language, I`ll take my chances in print statements instead reading assembly code..
I don't program in C but C++, and I will say that I do use std::cout for debugging a lot
ha, so that were the pesky files that were filling up my filesystem :) Thank you for this info!
It is a great video, no more crashing. I need only Mac version of it.