🌿 Week 16 Hobby Kernel Dev in C, x86: Physical memory allocator 🌿

(detailed timestamps in comments)
#livecoding #softwareengineering #operatingsystem
discord: / discord
Welcome! This is my chill Operating Systems development "gardening" project.
The "gardening philosophy" is to have fun, above everything else. This means taking baby steps and locking in small wins before building up for more ambitious projects. People also call this "hacking" on something.
My personal goal & vision is to build a minimal, high quality OS foundation which can be used for play, experimentation, and learning.
I'm personally not an expert systems programmer, although I'd like to be someday. Right now, but I'm just an intermediate one - I worked in infosec for 7 years, and music tech for 3 years. I've been exposed to the Linux kernel in both settings, but in highly constrained contexts, and mostly reading code, not writing it.
0:00 start
0:05: 🧠 Overview of implementing physical memory allocator in kernel development.
15:15: 🧠 Exploring the development of a basic memory allocator within a hobby kernel project
28:38: 💻 Exploring development of a physical memory allocator in an operating system kernel using C language.
37:44: 💻 Exploration of memory management in kernel development, including page directory setup and physical memory allocation.
49:24: 🧠 Exploration of physical memory allocation process and variable renaming in hobby kernel development
1:04:23: 🧠 Overview of physical memory allocation in kernel development
1:13:44: 🧠 Understanding the allocation and marking of physical memory pages in a hobby kernel development project.
1:25:20: 🧠 Exploration of memory allocation in kernel development, focusing on page list checks and modifications.
1:36:16: 💻 Exploration of memory allocation process in C, focusing on exclusion of memory regions and pointer manipulation
1:45:42: 🧠 Illustration of managing physical memory in kernel development, including tracking used pages and utilizing symbols for memory boundaries.
1:55:17: 🧠 Development process of a physical memory allocator in a hobby kernel project.
Recapped using Tammy AI

[@offlinemark]

0:00 start
0:05: 🧠 Overview of implementing physical memory allocator in kernel development.
0:05: Week 16 of kernel development stream, focusing on memory management
0:16: Preparing to work on memory management despite being sick
1:10: Moving on to memory management after completing lab one tasks
2:57: Tasks include implementing physical memory allocator and maintaining page data structures
4:24: Modifying files like M layout and pmap to define page structures
7:25: Examining existing kernel code and functions like Panic and assert
10:49: Implementing functions like boot Alec page and init page Alpage free
12:55: Confusion regarding specific code additions and function names
13:37: Focus on creating initial page directory before further modifications
15:15: 🧠 Exploring the development of a basic memory allocator within a hobby kernel project
15:15: Introduction to memory allocation for the kernel's page directory
15:53: Initial setup of a simple memory allocator using static variables
16:25: Functionality of the allocator to allocate contiguous physical memory
20:00: Description of the bump allocator approach being utilized
22:23: Implementation of memory allocation logic for requested block sizes
26:13: Testing and debugging strategies using print statements
27:09: Verification of allocator functionality through basic test runs
27:20: Initiation of print debugging to track variable values during execution
28:04: Preparation for executing the code to observe results and further debug
28:38: 💻 Exploring development of a physical memory allocator in an operating system kernel using C language.
28:38: The speaker is unsure about checking the character code for a specific item in memory.
29:35: They attempt to verify the consistency of the memory allocation starting point.
29:47: The speaker struggles with understanding how to check the BSS section for memory allocation.
31:53: A manual test is conducted to verify the memory allocation process.
32:11: The speaker considers implementing unit tests but decides against it for the current time.
33:22: An assertion is added to validate the memory allocation process.
34:07: The speaker intentionally triggers an assertion failure to test the validation process.
35:37: They express interest in developing their own operating system in the future.
36:35: The memory allocator functionality is successfully implemented and validated.
37:44: 💻 Exploration of memory management in kernel development, including page directory setup and physical memory allocation.
37:44: Discussion on using BSS section for memory allocation and kernel references.
40:06: Need for a structured approach to memory management in kernel development.
41:23: Creation of a new page directory for virtual memory layout.
43:05: Importance of setting up mappings for kernel execution and critical data access.
44:32: Importance of understanding virtual page tables for memory management.
45:00: Exploration of page table entries and their alignment considerations.
46:42: Use of global variables like PD Type for memory management operations.
47:01: Allocation of an array to track physical pages and corresponding page info.
48:49: Querying physical memory details through low-level hardware interactions.
49:24: 🧠 Exploration of physical memory allocation process and variable renaming in hobby kernel development
49:24: Discussion on available physical memory and variable values
54:05: Renaming of global variables for clarity
57:01: Allocation of memory based on page size calculations
59:53: Debugging process using the debugger tool
1:01:15: Verification of memory allocation through dumping and calculations
1:03:04: Confirmation of successful memory zeroing operation
1:03:36: Calculation of total allocated memory size
1:04:07: Discussion on zeroing out only the requested memory amount
1:04:23: 🧠 Overview of physical memory allocation in kernel development
1:04:23: Kernel development involves setting up initial data structures for physical memory allocation.
1:09:00: Allocating memory involves mapping memory using boot map region or page insert functions.
1:10:07: Kernel data structures include a list of free physical pages and pageinfos for tracking memory frames.
1:11:57: Pageinfos contain a refcount to track the number of pointers mapping to a page.
1:12:37: Intrusive linked list is used for storing pageinfos with direct links in the struct.
1:13:44: 🧠 Understanding the allocation and marking of physical memory pages in a hobby kernel development project.
1:13:44: Memory allocator functions are used to mark physical pages as free, with an exception for page zero.
1:14:05: Page zero is marked as in use to preserve certain structures, while the rest of the memory is considered free.
1:14:49: The reference count for each page is set to zero, even if already zero, during the marking process.
1:15:14: A pointer called PP link is used to maintain a linked list structure for free memory pages.
1:16:07: The process involves updating pointers and maintaining the integrity of the linked list structure for free pages.
1:19:04: The construction of the linked list is done in a way that appears to assemble it backwards.
1:23:14: The concept of 'in use' for a page is associated with preserving specific data structures within the memory allocation process.
1:24:42: The physical memory pages allocated using the page allocator have their pointers managed and updated accordingly.
1:25:20: 🧠 Exploration of memory allocation in kernel development, focusing on page list checks and modifications.
1:25:20: Page list checks are essential to ensure proper allocation and utilization of memory.
1:26:20: Understanding the functions and checks involved in verifying page entries.
1:27:25: Examining the calculation of physical addresses from page structures.
1:31:03: Identifying and asserting conditions for valid entries in the page list.
1:33:24: Initial modification to mark physical page zero as in use.
1:34:47: Consideration of exceptions in code execution based on page conditions.
1:35:13: Utilizing constants or arrays for better code readability and maintenance.
1:35:49: Iterative process of refining page list entries for efficient memory management.
1:36:00: Continued exploration and testing of modifications for desired memory allocation outcomes.
1:36:16: 💻 Exploration of memory allocation process in C, focusing on exclusion of memory regions and pointer manipulation
1:36:16: Discussion on excluding memory regions from allocation list
1:36:35: Implementation of excluding mechanism and potential impact on final shape of memory
1:38:02: Introduction of IO hole and its handling in memory allocation process
1:40:07: Conversion of physical addresses to indexes for memory allocation
1:42:36: Consideration of boundary conditions for memory exclusion and pointer manipulation
1:43:32: Simulation of memory allocation process with focus on IO hole region
1:44:01: Analysis of pointer swapping and list manipulation in memory allocation
1:45:09: Verification of correct pointer behavior for memory allocation list
1:45:42: 🧠 Illustration of managing physical memory in kernel development, including tracking used pages and utilizing symbols for memory boundaries.
1:45:42: Discussion on physical memory management and tracking used pages
1:46:59: Identifying kernel memory boundaries using symbols
1:54:07: Utilizing symbols for dynamic kernel size determination
1:55:17: 🧠 Development process of a physical memory allocator in a hobby kernel project.
1:55:17: Discussion on constructing page tables in the kernel binary
1:55:27: Allocation of page tables with boot allocator at compile time
1:56:09: Handling 'end' alignment issue in the code
1:57:24: Defining kernel regions and frame indexes for memory allocation
1:59:33: Marking boot allocator zone as non-free for memory management
2:03:12: Dynamic querying of memory allocation from the boot allocator
2:04:24: Debugging process to check page freelist functionality
2:06:14: Sanity checking variables and debugging incorrect calculations
2:08:44: Troubleshooting and adding print statements for further analysis
Recapped using Tammy AI

[@baibhabmondal1740]

Keep em coming! dont stop

[@offlinemark]

Thank you very much!

[@offlinemark]

highlight: figuring out page_init() bug
th-cam.com/users/clipUgkxDqATnxI0NaRVpx1gkbDeVjG7zYyQ0H3f?si=WFSxMO9npasVXmiE