Holy dang dude, you’re literally the most knowledgeable person about C libraries I’ve seen on TH-cam! You should be a teacher or something. Great videos, please keep em coming!
hello, been really enjoying your videos and learning a lot. i've got a question though: should we use munmap once we finished? kinda like free() is used after malloc(). and if so, is it the same for brk/sbrk (like can sbrk called to shrink or does it have a shrinking counterpart). I've done a bit of strace and found than munmap is in fact called by free() but haven't seen a counterpart with brk sbrk (or something like calling sbrk with a negative number perhaps as two's complement of the initially allocated memory)
I just stumbled upon one of your videos, and man, I went down a rabbit hole with your excellent guidance! Your explanations are so clear and thorough; Thanks for breaking down complex concepts into digestible bits. Your passion for teaching really shines through, and it's making a huge difference in my understanding. Keep up the fantastic work!
This is a fantastic video. One of the best you made yet. Strangely, it is one of the videos I watched last. There are two things that I feel have been left out a bit. First, when you showed that increasing the size of the memory being allocated dramatically made malloc use mmap instead of brk, you did not really tell us why that happened :) (I assume it happened because in the first case, there was enough left from the last page allocation, so all that was necessary was to raise the break?) Second, when allocating small chunks (in the order of bytes), I sometimes see multiple mmaps (with strace), but it always ends with brk, which means mmap is a higher level command and cannot function completely independently of brk. A bit of explanation would be super useful :) But regardless of all that, great video!
I once tried to write an "infinite storage glitch" before I learned that the address limit is not actually the physical limit. And of course, I tried to write it in python.
Hi Jacob, Thanks for your videos. I have one doubt here. In first program, void *second = sbrk(4096) ; here your are trying to increment to 4k bytes. But at 3:38 you are saying that "trying to increase 10byes but page size(4Kbytes) is rounded up. I couldn't able to relate.
2:10 According to the official brk manual, the program break is the first location after the end of the process's data segment and the uninitialized data segment. So if we look at the memory layout of processes we see program break isn't the top of the heap. Am I right ?
Hello Jacob. Thanks for your wonderful videos. I need clarification on the statement where you said adding 10 bytes further using sbrk would round it up to 4k as page size is 4k. I was playing with sbrk and see that adding 10 bytes and then checking the pointer address just shows it incremented by 10 bytes and not 4K, as given in the man page of brk. Is there a better way/example to understand this. Thanks CODE: void *s1 = sbrk(0); //to find the current brk void *s2 = sbrk(4096); //move the brk by 4K i.e. one page void *s3 = sbrk(0); //find the new brk void *s4 = sbrk(10); //move the brk by 10 bytes void *s5 = sbrk(0); //find the new brk printf("First: %p ", s1); printf("Second: %p ", s2); printf("Third: %p ", s3); printf("Fourth: %p ", s4); printf("Fifth: %p ", s5); OUTPUT IS : First: 0xbe5000 Second: 0xbe5000 Third: 0xbe6000 Fourth: 0xbe6000 Fifth: 0xbe600a
And, regardless of your answer. My comment refers to how modern virtual memory works. When you move the program break, it's effectively mapping more pages to the process's memory space. And, since pages are fixed size (usually 4KB), that means that the boundary between Ok and SegFault always increases by a multiple of the page size. Even if the OS implementation tells you it moved the program break by 10 bytes, the real boundary either didn't increase at all (because the needed page was already mapped) or it increased by 1 page (4096 bytes).
Hello Jacob I hope you are fine .. I just had a question .. how can I get the first and last addresses of data, heap and stack and dimensions for C language in my machine? (Linux)
So, if I `mmap` some memory as writable and executable, and then put some machine code in it, can I then jump into that code and execute it? :q Of course, it would be better to change that memory to read-only after I prepared the executable code in there. How can I change permissions on a pre-existing memory mapping? And how can I actually jump to that code afterwards? Is it enough if I make a fake function pointer type, and then use the address returned by `mmap` like this?: typedef int (*Func)(int x); Func *code = mmap(NULL, 4096, PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); /* populate it with some code */ /* change its permissions to PROT_READ | PROT_EXEC - how? */ int retval = code(123);
I get exact amount of bytes from sbrk. Not 4k aligned. Upd: I see it's aligned none the less: when incrementing less then a page and then writing into Third+1 memory, it doesn't segfault; but it does segfault when the increment is full page and thus Third is the real program break.
I'm not understanding the point of the program break, I've implemented a memory allocator before using mmap under the hood to get whole pages at a time and just linked them together with some pointers inside them, I never once used sbrk
3:34 I just tried to move BRK by some random num, like 42 with assembly code and syscall on linux x86_64, and it doesn't round it up to 4k, so maybe it's sbrk() internal's doing. I assume, the kernel does build up it by 4k increments, but it's transparent for brk caller or something
Thanks, rajan. Welcome. Can you be more specific? A page fault happens when you access a page in your page table that isn't resident in memory. You pay a penalty because the OS has to move the data into RAM from disk. You can use mmap to request pages of memory. How you use them will determine whether or not you have page faults.
@@rajanjha5797 I still think I'm misunderstanding. A Major Page Fault is when the requested page is not in RAM and the data must be read in from disk (it's slow but it's not an error). A Minor Page Fault is when the data is already in memory, but the page table needs to be updated (much faster, since no disk access is required). If the program tries to access memory that isn't mapped in the page tables, then you get a Segmentation Fault (sometimes called an Invalid Page Fault). You don't need mmap to create Invalid Page Faults. Just dereference NULL or any other unmapped address. I guess a rare case would be when you try to access a page that the OS thinks is out on disk, but then when it goes to the disk it can't find it. That would mean that either the OS messed up or maybe you had a hardware error (disk went bad or RAM failed). That shouldn't happen during normal operation-even with buggy programs.
@@JacobSorber You are right about it ...its a kind of segmentation fault but of different kind i recently came across of debugging such a situation but now i am not able to generate it ...maybe this link will able to help you understand that which i am talking about : smackerelofopinion.blogspot.com/2010/02/kernel-oops-page-fault-error-codes.html?m=1
Debugging page fault error is one of the popular question for interviews also would truly appreciate if you can try to explain such a topic as not much is talked about this over net ... :)
Hi Jacob, I'm experimenting with the same code as yours (using sbrk) to see what will happen if I access an address beyond the program break. But there is no segmentation fault. I'm using Ubuntu 18.04.
Did you use sbrk(0) ? Also, virtual addresses do exist beyond there - the globals and the code. But there is a MASSIVE gap - to allow expansion of the heap. But if you added a REALLY big amount you could end up in that area of the Virtual memory!
Thanks for the Precise introduction. As I know current computer has two size of page. Is that true?And how do I know my page size with what command or C code?
You can get the size of standard pages for your machine with getpagesize() on UNIX-style OSes. Not sure how it's done on Windows. As for the other size, I'm guessing you're talking about HugePages, which are usually 4-256MB and are handled a bit differently. Might be a fun topic for a future video.
@@shruthiabirami232 uint8_t is an unsigned 8-bit int. I have a rather old video about these types (th-cam.com/video/iX1uGr6Si0E/w-d-xo.html). As for, fd and offset, they relate to memory mapped files. More about that here (th-cam.com/video/m7E9piHcfr4/w-d-xo.html)
#include #include int main(void) { void *first = sbrk(0) ; // will tell where the memory break currently is void *second = sbrk(2) ; // move the break above/more by 2 bytes void *third = sbrk(0) ; // again tell where the memory break currently is printf("the memory break currently is in : %p ", first) ; printf("moved the break above/more by 2 bytes, new position : %p ", third) ; printf("trying to use beyond this region will lead to segfault ! ") ; int *ptr = (int *) third + 1 ; *ptr = 2 ; printf("%p ", ptr) ; return 0 ; } I wrote this, but it did not segfault, why ?
![[UNCUT]"ทนายตั้ม" แฉยับ! ความลับสวรรค์ ใครคือ "บิ๊กเทวดา" ของบอสพอล l คนดังนั่งเคลียร์ l 15 ต.ค. 67](http://i.ytimg.com/vi/ru9fpSaUwdA/mqdefault.jpg)
You should have also mentioned what malloc actually writes into the memory, which is why free doesn't need a size.
Ah, good point. I've been thinking that it would be good to do a more in-depth dive into how allocators work. I'll see what I can do. Thanks.
Your way of explaining things is one of the best that i've found on youtube! Thumbs Up!
Thanks.
Your way of presentation Awesome. Keep doing more videos. I subscribed.
Holy dang dude, you’re literally the most knowledgeable person about C libraries I’ve seen on TH-cam! You should be a teacher or something. Great videos, please keep em coming!
Actually if you click into his profile link, it says he is an associate professor of an univ in the states.
Found this helpful and enjoyable, thank you!
You're welcome. Glad it helped.
hello, been really enjoying your videos and learning a lot. i've got a question though: should we use munmap once we finished? kinda like free() is used after malloc(). and if so, is it the same for brk/sbrk (like can sbrk called to shrink or does it have a shrinking counterpart). I've done a bit of strace and found than munmap is in fact called by free() but haven't seen a counterpart with brk sbrk (or something like calling sbrk with a negative number perhaps as two's complement of the initially allocated memory)
What a crisp and clean explanation! I am impressed man. Thanks a ton.
You are my life savior......your every teaching helps me move a step ahead....you came late on TH-cam.....but at least you came......love you sir.....
Glad I could help.
@@JacobSorberSharing knowledge is a noble job......and you are pulling this off beyond good....thanks for everything.......
I just stumbled upon one of your videos, and man, I went down a rabbit hole with your excellent guidance! Your explanations are so clear and thorough; Thanks for breaking down complex concepts into digestible bits. Your passion for teaching really shines through, and it's making a huge difference in my understanding. Keep up the fantastic work!
please don't stop.
I'm not quite good in English but the way you present the INFO is something new and good.
PS: please don't stop.
Thanks. I'll try to keep things going.
Awesome! I've been looking for a good presentation of this topic for days. Subscribed!
I can't believe i'm only learning about this now, this seems like essential knowlage
amazing information , please more videos on mmap
wow, great explanation
This is a fantastic video. One of the best you made yet. Strangely, it is one of the videos I watched last.
There are two things that I feel have been left out a bit.
First, when you showed that increasing the size of the memory being allocated dramatically made malloc use mmap instead of brk, you did not really tell us why that happened :) (I assume it happened because in the first case, there was enough left from the last page allocation, so all that was necessary was to raise the break?)
Second, when allocating small chunks (in the order of bytes), I sometimes see multiple mmaps (with strace), but it always ends with brk, which means mmap is a higher level command and cannot function completely independently of brk. A bit of explanation would be super useful :)
But regardless of all that, great video!
Clean presentation!
Great brother. Keep on doing more..
I once tried to write an "infinite storage glitch" before I learned that the address limit is not actually the physical limit.
And of course, I tried to write it in python.
I think you will like this video: th-cam.com/video/JcJSW7Rprio/w-d-xo.html
Hi Jacob, Thanks for your videos.
I have one doubt here. In first program, void *second = sbrk(4096) ; here your are trying to increment to 4k bytes. But at 3:38 you are saying that "trying to increase 10byes but page size(4Kbytes) is rounded up. I couldn't able to relate.
2:10 According to the official brk manual, the program break is the first location after the end of the process's data segment and the uninitialized data segment. So if we look at the memory layout of processes we see program break isn't the top of the heap. Am I right ?
Thank you!
Hello Jacob. Thanks for your wonderful videos. I need clarification on the statement where you said adding 10 bytes further using sbrk would round it up to 4k as page size is 4k. I was playing with sbrk and see that adding 10 bytes and then checking the pointer address just shows it incremented by 10 bytes and not 4K, as given in the man page of brk. Is there a better way/example to understand this. Thanks
CODE:
void *s1 = sbrk(0); //to find the current brk
void *s2 = sbrk(4096); //move the brk by 4K i.e. one page
void *s3 = sbrk(0); //find the new brk
void *s4 = sbrk(10); //move the brk by 10 bytes
void *s5 = sbrk(0); //find the new brk
printf("First: %p
", s1);
printf("Second: %p
", s2);
printf("Third: %p
", s3);
printf("Fourth: %p
", s4);
printf("Fifth: %p
", s5);
OUTPUT IS :
First: 0xbe5000
Second: 0xbe5000
Third: 0xbe6000
Fourth: 0xbe6000
Fifth: 0xbe600a
Hi Neetish. What operating system (and version) are you using? Some may handle this differently than others.
And, regardless of your answer. My comment refers to how modern virtual memory works. When you move the program break, it's effectively mapping more pages to the process's memory space. And, since pages are fixed size (usually 4KB), that means that the boundary between Ok and SegFault always increases by a multiple of the page size. Even if the OS implementation tells you it moved the program break by 10 bytes, the real boundary either didn't increase at all (because the needed page was already mapped) or it increased by 1 page (4096 bytes).
A little confused. The man page you show says brk returns 1 on success but in your strace it's returning an address
Hello Jacob I hope you are fine .. I just had a question .. how can I get the first and last addresses of data, heap and stack and dimensions for C language in my machine? (Linux)
So, if I `mmap` some memory as writable and executable, and then put some machine code in it, can I then jump into that code and execute it? :q
Of course, it would be better to change that memory to read-only after I prepared the executable code in there. How can I change permissions on a pre-existing memory mapping?
And how can I actually jump to that code afterwards? Is it enough if I make a fake function pointer type, and then use the address returned by `mmap` like this?:
typedef int (*Func)(int x);
Func *code = mmap(NULL, 4096, PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
/* populate it with some code */
/* change its permissions to PROT_READ | PROT_EXEC - how? */
int retval = code(123);
Answering my own question again :J
The protection flags can be changed this way:
mprotect(addr, size, PROT_READ | PROT_EXEC);
Yes, more of these please !!
Will do. Anything specific you would like to hear more about?
@@JacobSorber maybe more about back end programming, processes, strace, signals, buffer overflows and alike.
@@hamamsofiane7098 Ok. thanks. I'll see what I can do.
Great content, as always. If you can answer me: could you explain how actual RAM dump actually works? I always wondered that bc of the Program Break.
your profile is awesome
3:40 0xDEAD is my new favorite hex number :)
Yep!
I get exact amount of bytes from sbrk. Not 4k aligned.
Upd: I see it's aligned none the less: when incrementing less then a page and then writing into Third+1 memory, it doesn't segfault; but it does segfault when the increment is full page and thus Third is the real program break.
This video is perfect.
I'm not understanding the point of the program break, I've implemented a memory allocator before using mmap under the hood to get whole pages at a time and just linked them together with some pointers inside them, I never once used sbrk
Can the stack size limit be modified at runtime or compile time ?
Thanks for the video
You're welcome. Glad you liked it.
Great video
3:34 I just tried to move BRK by some random num, like 42 with assembly code and syscall on linux x86_64, and it doesn't round it up to 4k, so maybe it's sbrk() internal's doing.
I assume, the kernel does build up it by 4k increments, but it's transparent for brk caller or something
What compiler did you use? Thank you
he uses clang
I liked this channel and have subscribed it..i have a question can you generate page fault error using mmap??
Thanks, rajan. Welcome.
Can you be more specific? A page fault happens when you access a page in your page table that isn't resident in memory. You pay a penalty because the OS has to move the data into RAM from disk. You can use mmap to request pages of memory. How you use them will determine whether or not you have page faults.
@jacob sorber Thanks for the prompt reply.. i mean to say "major page fault" its an error which occur when page is not found in disk also .
@@rajanjha5797 I still think I'm misunderstanding. A Major Page Fault is when the requested page is not in RAM and the data must be read in from disk (it's slow but it's not an error). A Minor Page Fault is when the data is already in memory, but the page table needs to be updated (much faster, since no disk access is required). If the program tries to access memory that isn't mapped in the page tables, then you get a Segmentation Fault (sometimes called an Invalid Page Fault). You don't need mmap to create Invalid Page Faults. Just dereference NULL or any other unmapped address. I guess a rare case would be when you try to access a page that the OS thinks is out on disk, but then when it goes to the disk it can't find it. That would mean that either the OS messed up or maybe you had a hardware error (disk went bad or RAM failed). That shouldn't happen during normal operation-even with buggy programs.
@@JacobSorber You are right about it ...its a kind of segmentation fault but of different kind i recently came across of debugging such a situation but now i am not able to generate it ...maybe this link will able to help you understand that which i am talking about : smackerelofopinion.blogspot.com/2010/02/kernel-oops-page-fault-error-codes.html?m=1
Debugging page fault error is one of the popular question for interviews also would truly appreciate if you can try to explain such a topic as not much is talked about this over net ... :)
Can you elaborate when and why this mmap is used?
If one can't read/write data to the space above the Program Break, that means, that he can't access Stack or can he?
I have been curious of the Thai books you have on your book shelf, what's that about :)
They're Khmer, but yeah, I lived in Cambodia for a few years and try to keep up with the language as best I can.
@@JacobSorber Your videos have really helped me crack some coding interviews. Thank you
Hi Jacob, I'm experimenting with the same code as yours (using sbrk) to see what will happen if I access an address beyond the program break. But there is no segmentation fault. I'm using Ubuntu 18.04.
Did you use sbrk(0) ? Also, virtual addresses do exist beyond there - the globals and the code. But there is a MASSIVE gap - to allow expansion of the heap. But if you added a REALLY big amount you could end up in that area of the Virtual memory!
Also if you allocated a local var AFTER reading the sbrk(0) - then you may be accessing that local var - which is valid in VM.
I using layered fascia uncle lenny& anty Ingrid
I am full scaled
Thanks for the Precise introduction. As I know current computer has two size of page. Is that true?And how do I know my page size with what command or C code?
You can get the size of standard pages for your machine with getpagesize() on UNIX-style OSes. Not sure how it's done on Windows. As for the other size, I'm guessing you're talking about HugePages, which are usually 4-256MB and are handled a bit differently. Might be a fun topic for a future video.
u r just awesome..i found u late..
Thanks. Better late than never. :)
what is first and second ? to what data type they belong to ?
They are (void*), so just pointers (memory addresses) with no implied type information about what they point to.
@@JacobSorber what is uint8_t ? can u explain what is fd and offset ?
@@shruthiabirami232 uint8_t is an unsigned 8-bit int. I have a rather old video about these types (th-cam.com/video/iX1uGr6Si0E/w-d-xo.html). As for, fd and offset, they relate to memory mapped files. More about that here (th-cam.com/video/m7E9piHcfr4/w-d-xo.html)
#include
#include
int
main(void) {
void *first = sbrk(0) ; // will tell where the memory break currently is
void *second = sbrk(2) ; // move the break above/more by 2 bytes
void *third = sbrk(0) ; // again tell where the memory break currently is
printf("the memory break currently is in : %p
", first) ;
printf("moved the break above/more by 2 bytes, new position : %p
", third) ;
printf("trying to use beyond this region will lead to segfault !
") ;
int *ptr = (int *) third + 1 ;
*ptr = 2 ;
printf("%p
", ptr) ;
return 0 ;
}
I wrote this, but it did not segfault, why ?
every tutorial is better with explosions
couldn't agree more.
Holy moly, that's why python and JS programmers are afraid of C!
C is awesome! And totally gives you freedom
FEED BEEF
666
Great video.
Thanks!