To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/fasterthanlime The first 200 of you will get 20% off Brilliant’s annual premium subscription.
Honestly, every one of your videos always feel like a blessing. It's reminiscent of a sage descending from the mountains to share his sacred knowledge with the village folk.
I see myself more like a fool tumbling down the hill every so often to bestow the curse of knowledge upon others, but I think the idea is roughly the same!
@@fasterthanlime You have an amazing ability to digest all you learned into a concise, interesting, and easy-to-follow manner. This video is excellent.
every one of your videos up until now has been right on the perfect edge of "I understand, but we're stretching it". I've got to be honest, this one managed to get over my head. That's not bad, its a challenge. Gonna have to rewatch it a couple times and do some extra research on top of it, I think.
Thanks for the feedback! That video definitely warrants at least a couple follow-ups, I have unscripted ideas that I think could be neat. Gotta change it up a little, those videos are an enormous amount of work.
This was absolutely fantastic. A follow up video or resources to explain things in more detail for people who don't have much low level experience would also be great.
@@fasterthanlime Again, fantastic work, you're a really great teacher. Watching it again, I wrote down things that I have questions on. There are definitely resources on most of things available and I have a good place to start learning from now. The largest gap in available resources to me seems like eBPF. What defines the memory sections (ro, rw, re)? (Making our own executable packer is on my list to read yet.) TLB makes sense for virtual->physical, but what does walking the page table mean? (Also how does the TLB cache recent memory accesses? Is there a large performance hit to TLB misses? And can the TLB only work for a single process at a time? How does flushing work?) What exactly is a page fault? What is stack/heap, and how to they relate to syscalls (break, mmap, madvise, etc)? Does madvise just allow the kernel to remove pages from the resident memory, but they're still in the virtual memory? What is malloc, malloc_trim, calloc, realloc, free, and how do they relate to syscalls, heap, virtual address space, etc? With your mention of eBPF I lost all understanding of what eBPF is. I thought that it was only for packet manipulation and statistics collection on network traffic. A video on eBPF would be awesome. Everything after ~18:00 I was completely lost. That could probably take a decade of learning. (ptrace, userfaultfd, what is a fd?, what's a setuid binary, self-tracing processes, PID namespaces, seccompbpf?)
Your content is the absolute nexus of production quality, content quality, and approachability. The thoughtful and smooth crescendo of complexity that keeps one engaged for the whole video/story.
"My previous video about this was, in fact, correct. I was as shocked as you are" I wish I had that confidence, but even more I wish I was at the level of saying such a thing.
I watched this video all the way through assuming that this was another big programming channel. I was surprised to see how small your channel is. There is so much quality!
"This town aint big enough for both of us" by "The Sparks" The one playing the piano is what this guy reminds me off. They were brilliant and so is he.
Your videos are great. I was looking for some rust videos when I stumbled upon them. I've been programming for about 20 years so most videos are painfully slow. Your videos always touch on subjects that I've never had to deal with. I've not learnt a huge amount of rust, but my understanding of how systems actually work has grown a lot, and enticed me to think more about the tools I take for granted every day and to think about in earnest how I would build them. So thank you!
Technically sections in an ELF don't need padding. The section header contains the size field, the loader in the kernel can just load the non-zero stuff and leave the rest. Filling ELF section with zeroes ensures that the page is also filled with zeroes, instead of garbage. It has nothing to do with page alignment. The virtual *adresses* still need to be page aligned though, as most MMUs don't support setting memory attributes on anything other than a whole page.
as always I'm gonna have to go over some of this stuff multiple times, but I'm learning so much. Thanks for making these videos Amos, I doubt I could clearn about this stuff without someone like you making it engaging.
You haven't looked at RR code base before? A lot of this stuff's in there, although, your project I think is a better source to learn this stuff from since it's probably smaller in scope (and it's in Rust, RR is written in C++). But you've got the sending of fd's over a socket, the injection of syscalls, etc and all that really cool Linux stuff.
I've never cracked rr open (but I've used it a couple times!) but I 100% trust you on that. mevi definitely makes for a smaller surface to study, and you can always drop into some of the crates it uses if you're interested in a specific abstraction!
I didn't expect the linux kernel to be so fiddly. Is everything related to low level OS that challenging? For example, memory management on windows? Because I got scared
I think the short answer is yes. But the good news is that it's "knowable", and then you slowly change from feeling intimidated by it to feeling irritated by it at times 🙃
This video exposes a lot of "false dogma" IMO. You traditionally hear the simple explanation of the cost of dynamic vectors, usually saying something like "if a push exhausts the capacity, you have to reallocate and copy everything over", which is *morally* true, and maybe was *actually* true 20 years ago. Nowadays, on modern OSes its interesting to see how it's not the case. Yet another reason to never preemptively optimize. Measure first!
I love these videos! I’d happily watch a one or two hour version. I have ADHD, so your articles are a lot harder for me, but these videos are perfect!!!
Oh, these great contributions of yours. You *do* have an impact on the way I think about- and write Rust. Looking forward to some eBPF. Will you go the Aya way? 🤓🐝
It's so crazy that the minimum memory a program can allocate (4k) used to be sold as high end ram sticks. I was reading a comment on some other tech video and a person said their computer could have up to 20 4k ram cards. But that would take all their expansion slots.
It was another time, for sure! From devblogs.microsoft.com/oldnewthing/20200728-00/?p=104012 Huge pointers could refer to memory blocks larger than 64KB by adjusting the segment whenever the offset overflowed.¹ Pointer arithmetic with huge pointers was computationally expensive, so you didn’t use them much.²
damn, ok, growing a Vec looks a LOT nicer than something like ArrayList in JVM, which I think DOES copy the array. - I might be wrong and it does some magic behind the scenes, but this looks beautiful ...
I wonder if this needed so much unsafe code, it would have been easier to write in Zig ? Probably not, but I wonder. (and yes, it would probably not share code with browser) - Also, I have zero clue how one would even research all the documentation to make all this crazy stuff work ... - How to even figure out all the magic kernel stuff? And how to do it from something other than C ? EDIT: 22:22 - I see. So that's how! Naruhodo.
I've seen that same "we switched from unsafe Rust to Zig" article that you have, but in that case, no, Rust was plenty comfortable, thanks to a wealth of crates that exposed all the system interfaces I wanted :)
This made me think about a video about the same topic I saw a while ago, there they made a tool that compacts the fragmented memory pages of a process. Might be an interesting watch th-cam.com/video/c1UBJbfR-H0/w-d-xo.html
I have quite a few videos about that but I've been slowly figuring out how to do more accessible videos with a tighter scope. I'll get it right someday!
To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/fasterthanlime
The first 200 of you will get 20% off Brilliant’s annual premium subscription.
Yes!! Go get that sponsors boyy
Stack that paper baby
Honestly, every one of your videos always feel like a blessing. It's reminiscent of a sage descending from the mountains to share his sacred knowledge with the village folk.
I see myself more like a fool tumbling down the hill every so often to bestow the curse of knowledge upon others, but I think the idea is roughly the same!
@@fasterthanlime You have an amazing ability to digest all you learned into a concise, interesting, and easy-to-follow manner. This video is excellent.
every one of your videos up until now has been right on the perfect edge of "I understand, but we're stretching it". I've got to be honest, this one managed to get over my head. That's not bad, its a challenge. Gonna have to rewatch it a couple times and do some extra research on top of it, I think.
Thanks for the feedback! That video definitely warrants at least a couple follow-ups, I have unscripted ideas that I think could be neat. Gotta change it up a little, those videos are an enormous amount of work.
PHD-esque stuff. Mind-blowing 🤯
This was absolutely fantastic. A follow up video or resources to explain things in more detail for people who don't have much low level experience would also be great.
I have a couple in mind, but can you give me a few areas in particular you'd like cleared up?
@@fasterthanlime Again, fantastic work, you're a really great teacher. Watching it again, I wrote down things that I have questions on. There are definitely resources on most of things available and I have a good place to start learning from now. The largest gap in available resources to me seems like eBPF.
What defines the memory sections (ro, rw, re)? (Making our own executable packer is on my list to read yet.)
TLB makes sense for virtual->physical, but what does walking the page table mean? (Also how does the TLB cache recent memory accesses? Is there a large performance hit to TLB misses? And can the TLB only work for a single process at a time? How does flushing work?)
What exactly is a page fault? What is stack/heap, and how to they relate to syscalls (break, mmap, madvise, etc)? Does madvise just allow the kernel to remove pages from the resident memory, but they're still in the virtual memory?
What is malloc, malloc_trim, calloc, realloc, free, and how do they relate to syscalls, heap, virtual address space, etc?
With your mention of eBPF I lost all understanding of what eBPF is. I thought that it was only for packet manipulation and statistics collection on network traffic. A video on eBPF would be awesome.
Everything after ~18:00 I was completely lost. That could probably take a decade of learning. (ptrace, userfaultfd, what is a fd?, what's a setuid binary, self-tracing processes, PID namespaces, seccompbpf?)
Your content is the absolute nexus of production quality, content quality, and approachability. The thoughtful and smooth crescendo of complexity that keeps one engaged for the whole video/story.
"My previous video about this was, in fact, correct. I was as shocked as you are"
I wish I had that confidence, but even more I wish I was at the level of saying such a thing.
Thats not co fidence though...
It's such a relief that people like you make content for us like this. This is exceptional, cheers.
I watched this video all the way through assuming that this was another big programming channel. I was surprised to see how small your channel is. There is so much quality!
"This town aint big enough for both of us" by "The Sparks" The one playing the piano is what this guy reminds me off. They were brilliant and so is he.
Your videos are great.
I was looking for some rust videos when I stumbled upon them. I've been programming for about 20 years so most videos are painfully slow.
Your videos always touch on subjects that I've never had to deal with. I've not learnt a huge amount of rust, but my understanding of how systems actually work has grown a lot, and enticed me to think more about the tools I take for granted every day and to think about in earnest how I would build them.
So thank you!
I can't believe the "this video is sponsored by" bit actually paid off
It was all just practice 🧐
Love this content. Thanks for all the work you put in to explain this so well
Technically sections in an ELF don't need padding. The section header contains the size field, the loader in the kernel can just load the non-zero stuff and leave the rest. Filling ELF section with zeroes ensures that the page is also filled with zeroes, instead of garbage. It has nothing to do with page alignment.
The virtual *adresses* still need to be page aligned though, as most MMUs don't support setting memory attributes on anything other than a whole page.
great video, as always :)
btw. nice seeing you have a (real) sponsor now!
as always I'm gonna have to go over some of this stuff multiple times, but I'm learning so much. Thanks for making these videos Amos, I doubt I could clearn about this stuff without someone like you making it engaging.
While way over my head for now, this feels amazing! One day, I might be able to understand what's going under the hood.
Congrats on getting sponsored!
I love the visualization of memory allocations
What's crazy is I looked this up earlier this morning, taking a break now and watching youtube I see this lmao
You haven't looked at RR code base before? A lot of this stuff's in there, although, your project I think is a better source to learn this stuff from since it's probably smaller in scope (and it's in Rust, RR is written in C++). But you've got the sending of fd's over a socket, the injection of syscalls, etc and all that really cool Linux stuff.
I've never cracked rr open (but I've used it a couple times!) but I 100% trust you on that. mevi definitely makes for a smaller surface to study, and you can always drop into some of the crates it uses if you're interested in a specific abstraction!
Thank you Brilliant for sponsoring this video ❤
Crushed it again Amos! So many things to try this out on!
Fantastic video, thank you for making it.
I'd love to see a video of using mevi to optimise existing programs.
I'm so happy that you casually use the term "kibibytes" :D
(I had this discussion a few times with unteachable people...)
I didn't expect the linux kernel to be so fiddly. Is everything related to low level OS that challenging? For example, memory management on windows? Because I got scared
I think the short answer is yes. But the good news is that it's "knowable", and then you slowly change from feeling intimidated by it to feeling irritated by it at times 🙃
this video was great, I loved the introduction.
This video exposes a lot of "false dogma" IMO. You traditionally hear the simple explanation of the cost of dynamic vectors, usually saying something like "if a push exhausts the capacity, you have to reallocate and copy everything over", which is *morally* true, and maybe was *actually* true 20 years ago. Nowadays, on modern OSes its interesting to see how it's not the case. Yet another reason to never preemptively optimize. Measure first!
Excellent job, and congrats on the sponsor!
Amazing video, like everyone else mentioned, absolutely mind blowing stuff! Keep up the great work and congrats on the sponsorship!
Not sure I understood any of that, but I was impressed ;)
Super cool! I love the observability, and this is the next level!
I love these videos! I’d happily watch a one or two hour version.
I have ADHD, so your articles are a lot harder for me, but these videos are perfect!!!
You're a good teacher. You explain things differently, even a 2 years old kid will understand anything you explain.
rm -rf = rename --recursive --flawlessly
Basically solves the slash problem in all files
Blown away by every single one of these videos
That was both interesting and very enjoyable to watch! Thanks a lot for making these videos.
yo brah it was amazing, can't wait to see the eBPF :)
Yooo get that bag! Good job Amos!!
Really enjoy content like this
Another fire fasterthanlime video let’s GOOOOOOOOO
Have nothing to say, just commenting to help this video out so more people can see it :) insanely high quality
this is pretty cool! Your profile pic reminds me of the scrap metal artist from the Iron Giant
Could you please share your code editor setup? I saw you using VSCode and vim extension, what other configs/extensions do you use? Thx!
Oh, these great contributions of yours. You *do* have an impact on the way I think about- and write Rust.
Looking forward to some eBPF. Will you go the Aya way? 🤓🐝
Or no. It won't be the Aya way. Just as likely you're gonna roll it yourself 😋
I've naturally looked at Aya since I've used it before for networking, but I'm unsure whether it'll do the job for that project. Time will tell!
Thanks for captions!
Thanks! I used Whisper for these but even then it's a solid 2.5x-the-runtime of manual cleanup/sync work :)
17:29 Don't tell me what to do magic mind reading slides!
Wow, great video. Im glad i found this channel.
@fasterthanlime , you get mad respect.
great video thank you.
It's so crazy that the minimum memory a program can allocate (4k) used to be sold as high end ram sticks.
I was reading a comment on some other tech video and a person said their computer could have up to 20 4k ram cards. But that would take all their expansion slots.
It was another time, for sure! From devblogs.microsoft.com/oldnewthing/20200728-00/?p=104012
Huge pointers could refer to memory blocks larger than 64KB by adjusting the segment whenever the offset overflowed.¹ Pointer arithmetic with huge pointers was computationally expensive, so you didn’t use them much.²
If only my OS class was this good. ❤
At the beginning of the video, I thought Brilliant's starting a Rust course!
Great video. What's the website you use for making those sketches at @3:11 for example?
That's draw.io / diagrams.net!
Going from "sponsored by coffee" to "sponsored by Brilliant". TH-cam achievement unlocked.
All that caffeine finally paid off!
This is great stuff! I like your sense of humour.
How do you get the memory management sounds? I want one of those. :)
Haha that's all editing magic! I took the sound for StoryBlocks (I had a subscription back then.)
so, do we see if Firefox is leaking memory in the "third way" ? That would be a cool thing to find out after the effort.
damn, ok, growing a Vec looks a LOT nicer than something like ArrayList in JVM, which I think DOES copy the array.
- I might be wrong and it does some magic behind the scenes, but this looks beautiful ...
Damn... Great video man.
Incredible video as usual
This is not a trivial subject. Good work overall
Your doctor is great!
Buses and trains don’t exist in that universe
I didn’t know you visited America recently :o
very interesting, as always!
Missed opportunity here - should have been built on Gentoo or at least Arch 😉
In all seriousness - massive thumbs up 👍
Oh I've served my time in both the Gentoo & ArchLinux jails, I'm free now, I can use Ubuntu and still have street cred 😎
Please do a series on Axum indepth.
Very interesting, thanks for sharing!"
Yay sponsorship!
Your videos are always mind-blowing, not only for their content but also for the quality of the presentation. Kudos!
could you make a thing like RemedyBG... but for linux?
This is way above my pay grade.
Rly cool, keep up the content, it's really interesting l!
Can you do Zig vids as well
Thanks!
Thank you for your support !
I’d like to see faster than lime’s car.
I wonder if this needed so much unsafe code, it would have been easier to write in Zig ? Probably not, but I wonder. (and yes, it would probably not share code with browser)
- Also, I have zero clue how one would even research all the documentation to make all this crazy stuff work ...
- How to even figure out all the magic kernel stuff? And how to do it from something other than C ?
EDIT: 22:22 - I see. So that's how! Naruhodo.
I've seen that same "we switched from unsafe Rust to Zig" article that you have, but in that case, no, Rust was plenty comfortable, thanks to a wealth of crates that exposed all the system interfaces I wanted :)
@@fasterthanlime hehe, yup. - Awesome to know! - It's good to know it can be abstracted away into a safe crate.
Everytime you say "Mevy" I hear "Bevy" instead (rust game engine)
Gold
imagin u#r talking about linux kernel very basic/complex stuff and get an normal sponsor xD, bro u made it xD
lol "Stop reading these slides and pay attention"
AMAZING
Keep making more videos pls 🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏
This made me think about a video about the same topic I saw a while ago, there they made a tool that compacts the fragmented memory pages of a process. Might be an interesting watch th-cam.com/video/c1UBJbfR-H0/w-d-xo.html
Yes! I've been meaning to talk about MESH - anything Emery Berger is involved in is sure to be fascinating :)
nice
😨 18:13
I have no idea what this man says, but its very uhm.. interesting?
"a lesser man"
Pagliaci joke😂😂😂
For a month? Hummm...
I don't understand 7 in each 10 words you say.
Is my pronunciation that bad?
@@fasterthanlime absolutely not. I have a little bit knowledge about low level programming. The topics looks so deep that I can't get it.
I have quite a few videos about that but I've been slowly figuring out how to do more accessible videos with a tighter scope. I'll get it right someday!