In-depth: ELF - The Extensible & Linkable Format
ฝัง
- เผยแพร่เมื่อ 1 มิ.ย. 2024
- ELF files are everywhere: From your PS4, to the Wii, to any Linux device out there. In this video we explore the Extensible & Linkable Format, and understand how it works - on the byte level!
Errata:
- At the beginning I say extensible while displaying executable and linkable file format - Extensible is the old name, nowadays ist called Executable. Must’ve gotten confused :)
Further reading:
- The ELF specification: refspecs.linuxfoundation.org/...
- elf.h from the Linux kernel: elixir.bootlin.com/linux/late...
- How programs get run: lwn.net/Articles/631631/
- TLS: docs.oracle.com/cd/E19120-01/...
- Relocation: refspecs.linuxbase.org/elf/ga...
- sstrip: github.com/BR903/ELFkickers/
Timestamps:
0:00 Intro
0:47 segment and section differentiation
2:05 ELF structure
3:39 ELF header data structure
9:34 Program header data structure
13:54 Section header data structure - วิทยาศาสตร์และเทคโนโลยี
Incredible how a format developed so much time ago is still capable of being fully functional without any need of a version change. That tells a lot on the foresight of the creators!
I agree. The Unix design is quite elegant, some have called it timeless.
WebAssembly is another format that I think is well designed and will last well into the future.
Also very apparent that C and UNIX are conceived in parallel. The memory space is very much what C programmers are taught
ELF is not that old, it was adopted in 1999 as a replace for the old and simple aout format.
@@TiagoNateldeMoura I'm not sure what you mean. Linux and other *nixes was not using ELF from the beginning, but it was introduced in 1988 with System V 4.0
The difference between the program headers on 64-bit is for layout. The fields in the elf headers are aligned to their natural alignment, so 8-byte types are likely going to want to be aligned to 64-bit. It was done to move the 32-bit members together so the header wouldn't have to contain unnecessary padding.
LOL I just watched the video, saw the part where he explains the difference and immediately had the same idea. I paused, thinking "I wonder if..." and found your comment right away. :-)
@@Colaholiker lol same. cheers
I asked chatgpt before reading the comments lel
good to know it was correct
0:47 segment and section differentiation
2:05 ELF structure
3:39 ELF header data structure
9:34 Program header data structure
13:54 Section header data structure
Thanks a lot, I put it in the description!
@@stacksmashing Ik it's an old commend, but It'd be nice if you added a 0:00 as well, so youtube would automatically add the chapters to the video player
@@Dwyriel Ahh thanks that note! Fixed!
13:00 Ah, the infamous `p_align` field that *everyone understands wrong* :P This is _not_ about 4-byte vs. 8-byte alignment (and you would know that if you ever tried hand-crafting ELF files or fiddling with existing ones), but it is meant to facilitate *paging* . Many people say that segments have to be aligned to page boundaries, but *this is also wrong!* The only true requirement is that their _offsets_ in file and in memory must be *congruent modulo the `p_align` value* (which is usually some reasonable multiple of the page size). The reason for this is that when segments from the file are memory-mapped (using `mmap`), the granularity of that mapping is the page size (i.e. you can only map whole pages), but the address is usually assigned by the kernel, and it might change on different runs. This isn't a problem, though, as long as data and instructions keep the same offsets from the page boundary. Then they can be easily mapped to different addresses in different address spaces, and yet their offsets from page boundaries stay the same as they were in the segment in the file. If the offset in the file were different than in memory, the system loader would have to shift it so that the offsets matched again, but then the beginning of the segment in memory would have to cross the page boundary, so the loader would have to map an additional page in front of our page, with lots of padding at the beginning. It would impact performance, memory usage, and be much more complicated to implement. Hence the alignment criterion that makes sure that when segments are mapped into memory, the offsets in the file and in memory match each other (i.e. they're the same modulo `p_align`).
13:16 This is because of machine word alignment. Two `Word`s are 64-bit together, so the next field will be aligned to 64-bit machine word boundary. Otherwise a padding would be needed which would only waste space. But since this padding is precisely the size of `p_flags`, they decided to rather move the `p_flags` field in there than waste this space for paddings.
Thanks for the insightful comment! Much appreciated
Most pedagogic content about a presentation of how the ELF format work.
Top tier quality content
Great to see you're back!
Please keep making videos like this, all of your videos are so helpful and the topics are explained so well! You are awesome man, I have not absorbed info like this in a while.
Very concise and precise overview of the ELF format. One of the best, if not best summary of the elf, program and section headers
Awesome tutorial! Haven't looked into what happens inside an ELF since the time when Amigas used 68k and PowerPC simultaneously!
Very nice explanation! Definitely helpful and clearly explained! I would love to see/hear more from the reverse engineer's perspective like when you demonstrated the "sstrip" tool and that an ELF can still run without section headers. In other words things that we take for granted in a normal workflow but during reversing they may appear differently.
Awesome Work. I really enjoy these deeply technical videos, keep up with them.
i just started writing a bootloader and this is exactly what i need :)
What is your bootloader doing with ELF? I am genuinely interested.
@@homelessrobot Probably stuff like loading kernels, as you can use ELF for that
Well, I guess it's a second stage bootloader then.
This video was so well made and thought out! Kudos! :)
Thank you for the well presented, well written video. Keep up the good work!
Great video! Happy to see you back.
Very well presented video, it would be interesting to see more details on the symbol table and how it compares to PE binaries and pdb files on Windows.
Underrated channel, it's an awesome video (very useful for the ctf i m doing right now xD), great job ! Can't wait for other videos.
Thanks for a hugely informative description. I've been working with elf files for years without fully understanding the structure. Great job.
Glad you back, thanks for the infos
Great explanation! I love digging into executable formats... wrote my own disassembler and have PE/COFF/ELF/etc loaders. My Elf reader code converts the ELF32 structures to ELF64.
Very nice explanation, clear, and directly to the point! Thank you!
Nice video! Thanks also for the further readings!
Oh, acturally ELF is *not exactly* the executable format of the PS4,
Rather 'SELF' which is like ELF but with some extra stuff added at the start mostly some boot flags, authentication & permissions info if its unencrypted (only on devkits) etc and then the actural elf header and stuff
Though it's often pretty easy to strip away the SELF stuff (and sometimes decrypt its contents too..) which basically just converts your SELF into a standard ELF binary- but the console cannot run ELF directly !
Oh also SELF is used on the PSVita and PS3 as well. and the format of SELF is different on all there consoles..
Lol on PSVita SELFs acturally have 2 ELF headers lol not sure about PS4 though.
Only PS2 and PS1 use ELF directly!
As far as I know the PS1 uses EXE's, they are kind of based off the ELF format, but simpler. The PSP does use plain old ELF's, iirc.
@@ederbarrero5585 nope PSP is self too (see ~PSP header) on UMDs or its PBP for digital stuff
@@LiEnby Ah, you are right, I forgot the PSP does have an encryption scheme for its executables too.
Welp, that only leaves PS2 in the "it runs ELF files directly" gang.
That's how I was able to load Free McBoot onto one of my PS2 memory cards without having to `hard-mod` the system! It is a process though as you need a specific game title that you have to rip from disc using specific software to a computer, then you need different software to modify the copy, and another piece of software to patch the modified copy, a dvd burner with a specific program to burn it properly. It does involve taking the PS2 apart and removing a spring, and blocking a couple of push buttons to stop indicating that the tray is opened. This works for the "slim" model only, there are other methods for the fat model... Then you will need a blank memory card, a USB thumb drive less than 1GB preferably 256 - 521 MB as the USB ports on the PS2 only support USB 1, maybe USB2 with some of the newest versions of the slim models... and they don't support large size drives that you see today such as 8-256 GB. You will then have to play that specific title up to a certain point, save the game at a certain point, and then when you go to proceed to another game state such as a loading screen to the next level, you have to pop the original disc out with the tray open and while the system is still spinning because it believes the tray is closed, you will then have to pop in your burnt version of the game that will have the `elf` file that you added to that game disc within a specific track and sector... Then if done properly, the console will read that `elf` file and execute it. This acts like a `flash bios operating system - file manager`. Now with this elf running, you can now insert your flash drive that has other elf files... It's been a while since I've done this, but you either move them over to the memory card directly, or you "run" the Free McBoot directly from the USB as it will format your memory card, you can also have other applications and utilities on that USB such as media players, NES, SNES, Sega Emulators and Roms, and more. Once you have Free McBoot installed onto the memory card, you can then shut the system down, remove the discs and USB drive... start the system back up, with the modified memory card. Now, when your PS2 loads the main menu that you would see without any discs have changed. You can customize this list of applications with the same elf file that you used from the modified game disc, however, you will have a copy of this on your memory card so you won't have to do the hot disc swap trick. From there I was able to install, I believe it's called OPL2 or something close to that... and I configured one of my hard drives on my PC to have `ISO` images of all my game discs. This is another long process to setup... but I can now run any of my PS2 and PS1 games from computers had drive across my network. This process is involved too... Even though I am a Windows user when it comes to PCs mostly due to "Gaming" reasons and that most of my systems came with Windows... Now, I don't mind Linux, but I can't stand Apple! But yes, that is the power of ELF unlike Window's EXE!
@@skilz8098
Interesting.
You should upload a tutorial!
Looking forward for the next videos!! great job :)
Nice Video and Awesome Explanation thank you so much :)
Great explanation, thank you. Definitely keeping this in my favorites list
Great video !!!! Looking forward to your update of ELF!!!
great work. it really helps to understand elf
Amazing . Densely packed , saved a lot of time
the editing style reminds me of retro games mechanics explained! nice explanations, though maybe a touch too fast-paced in explanations
Glad you liked it! Yeah, it's definitely difficult to find a good pace :)
I liked the pace
Watching this again for recap 2 years later now because the title image got me... Extensible was the old name, now it's Executable, and you even mixed both here. Wow 😮
Thank you so much for this awesome explanation! Appreciate it!
Please do a video about relocations, is such a important topic about ELF format! That one was pretty good walkthrough.
nice animations and explanation - must have taking a long time to make 👍
Hello to ITMO students, who are trying to write an elf parser for Risc-V. And thanks for an excellent video explanation.
exemplary presentation. smashing for president ! thx for sharing, have fun, aum
This is excellent! Thank you so much!
Very great explanation, thank you ❤
Thank you for this awesome video.
One of the few youtube videos where you have to reduce the playback speed. Great Video !!
Great tutorial! I learned a lot.
Really interesting video!
If anyone is interested in learning about the execution of programs in Linux, I recommend a series of articles by LWN titled 'How programs get run', and its follow up 'How programs get run: ELF binaries'. It explains how the OS loads executables into memory and how it invokes an interpreter on them if needed (eg. the shell for scripts, or the ELF program loader for dynamically-link programs).
Much needed info. Thanks for the tip!
hopefully when I google LWN ?
Subscribed!
always a good day when you learn something new.
Just curious - if sections are only used during compiling, and removing them doesn't affect the execution of the program, why aren't they stripped out as part of the compile process?
(I'm presuming this only applies to executable ELF files - unsure if an object or library still needs these, but I can kinda see that they would...)
Amazing video!
Great video !
I'm developing my own operating system, and this is EXACTLY what i need.
Might yoink elf for my weird graphing calculator bytecode virtual machine thing.
Thank you! Very cool video!
Nice video i literally just started studying about executable file formats and then i see you uploaded this.
Will you do one about the PE format next? I heard its kinda complicated compared to ELF so it would be a big help
Wow cool nickname (even screws with the site layout a little bit). How did you do it? (Especially the line break)
@@sillysimon7889 Thanks, its pretty easy you just put this character: "⸻" a few times after your username. Even though youtube has a character limit to prevent long usernames, i guess they didn't consider Unicode characters that are wider than a single character. There is no newline character, I think the linebreak just happens because its the default behavior of the browser to split it like this when it cant fit it all in the container.
You can actually do all sorts of weird things with unicode characters for examples: M̵̶̛̙̟̫͇̪̖̯͕̩̝E͏̡̢̤͈̩͍̯̗͕͎͎̥̫̘̙͖͇̤̻͢͜ò̦̳͈̰̩̣̙͇̖̭̪̲͖̤͡͝ͅͅͅn̷͉̼̱͉͎̜̰͜_Ȩ̷̝̦͕̮̫͇͖̪̪̀͟͝ͅ ̶̱̳̼̝͖̹̹̻̣̺͓̜͜͞͡Ś̴̢̛̥̪̭̳̥̞̩̞̙̼͍̟̗̻̮͟A̵̛̮̪̗͎̺̼̯̠̘͉̼͠ͅi҉̩͔͖̳̰͕͉͕͓̘͚͙͔̰̦͔̜͘͝ͅd̥͔͖̥̠͎̻͍̣̯̞̗͇͜͜ ̕͞͏̰͕̠a̴̭̟̠̞̳͇̙͕̻̫̘̭͈̲̹̕͠ņ͉̣̻̩͜_ṇ̵̢̼̺͎͟ͅó̡҉̼̫̳͙̰͔ͅÝ̶̲̞͇̪͈̥̮̫̹͕͡Į̶̛͖͕͔̩͖̼_N̷̜̩̜̤̲̬̞̯͘͜G̟͍͔̹̣͈͢͡?̡̹̹̟͖̕͟͜͠ Z̷͙̗̻͖̣̹͉̫̬̪̖̤͆ͤ̓ͫͭ̀̐͜͞ͅͅα - the only problem is not all unicode characters render on all platforms but i've found the 3 dash character in my username seem to work on most
Yeah I agree too. A video on PE file format would be great help.
PE is Windows-only. It’s not used on any other platforms.
@@lawrencedoliveiro9104 thats actually exactly why im so interested in it. So many closed source programs are windows only and so knowing the windows executable format would be helpful in reverse engineering. Most ELF programs are open source anyway :p
very good explaination.
Thank you! I'm trying to make a compiler and this is very helpful. The System V spec is very detailed and it's not the best thing to start up for beginners.
It's also used in PS2 games and PSP games. Both the main executable on the PS2 disc and the boot.bin (once decrypted) on PSP umds are standard ELFs.
Long time .
Learned alot .
Next PE please .
I’d love to see a follow up video for this on DWARF. Lots of fun Turing completeness to be had
awesome explanation
great video :)
The order of the ProgramHeader is surely (my guess not confirmed knowledge) different between 32 and 64 so that the data aligns to 32 respective 64bit boundaries. For 32bit you have: Word = 32, Offset = 32, ... so all is fine, but for 64bit that does not work: Word=32, Offset=64. Thus, you want to have: Word+Word=32+32=64, Offset = 64. I guess the 32bit version wasn't changed to not break backwards compatibility
Interesting theory, that might be it 🤔
I was reading the Practical Binary Analysis by Dennis Andriesse when this video arrived...
oof, the intro shows the E being "Executable", but in the title, it's "Extensible". Voice says "Extensible". Executable is right though lol
Oof, thanks! It used to be called Extensible, must’ve gotten confused eventually :)
@@stacksmashing Oh, seems like it actually was formerly called Extensible, but now it's Executable.
The last time I watched your videos you were ghidra ninja
here's something I'm stuck on: if a section describes where in virtual memory it is mapped to, then this implies that the creator of the ELF has some assumptions about virtual memory, for example its size. Do architectures have specified virtual memory sizes? For example, setting e_machine to EM_ARM indicates the ELF is for an ARM platform, but depending on the architecture isn't the VM size different?
10:58 you said that only executable will have PT_INTERP, but in fact, the shared libraries (e.g. libc.so) also have PT_INTERP because they may link other libs as well. Correct?
14:38 I think the command is to remove the section headers, not the content it pointed to, because the content is shared with the segment, i.e. it's needed when segment is executed.
Have you looked at how, say, it compares to COFF? I recall that was popular among some Unix vendors back in the day. IBM (of course) created their own variant, called XCOFF, which Apple adopted for use on the PowerPC Macs.
an extensible video about the extensible and linkable format
PT_SHLIB is in fact in the spec. It's just undefined as of right now, so you can do whatever you like with it, but "here be dragons".
In that, there's no guarantee that you'll stay conformant if it changes. So yeah.
3 years waiting for the followup!
Is there any in depth article or anything which explains the position independent execution in depth?
13:18 looks like it was made for better alignment of struct. that way it would require less space since you don't need alignment before next Xword field.
Wii stuff on the middle and end of it's generation moved from elf to dol, also iirc the gamecube also uses elf sometimes
0:56 Actually both words have slightly different meaning and are being used logically. Think of it like this "Segement of TV show" or "Section of TV show" that is exactly why they used "segment" for the runtime.
That was good, now please do as many other file formats as possible (executables or not), including proprietary ones
Godsend
The Force is strong with this one.
Why can't I read the segments and sections using their c struct from withing the program?
holy moly this is complicated
KDE has a new program called ELF Dissector.
Please review it!
I didn't understand the part of et_exec : it is doesn't support position independsnt executables? That is weird because I can compile any program without -no-pie and I can execute it on my system!
The type ET_EXEC only supports position dependent code. So when you compile with -no-pie the ELF file will be of type ET_EXEC. If you compile with PIE it will be ET_DYN.
I'm confused by the section header address field. I though sections are not used for execution. Why should it appear in "the memory image of a process"? What is a memory image? If it unused, wouldn't it be a waste of time and resources to load it into memory?
Maybe sh_addr is used to connect the section with the segment that has been mapped on to the same memory.
It would be good to see a video on reverse engineering a React Native app for Android, iOS, or both. Especially one that just uses a WebView. Should be very easy but I'm struggling.
Intresting
What is different between e_version and EI_VERSION?
ELF was also used in the ps1 and ps2 afaik.
A few time you mentioned some differences between 32 and 64 bit. A bit sad that you didn't said how it is on other systems, like 16 bit and 8 bit. My MSP430 is a 16 bit processor and my AVR is a 8 bit processor, i use GCC and ELF-Files for both of them.
And they are either 32 bit or 64 bit ELF files :) it says nothing about the contents of the sections, but about the addressing used in the file.
@@stacksmashing Thank you, i didn't know that. It would probably help to mention that because it is easy to confuse ELF bit width with architecture bit with.
Hows the Nintendo game & watch coming along?
So, basically, its a messy context-sensitive grammar? Or is ELF actually Turing-complete?
Didn't expect to find another pony here :) /)
As for your question: I don't think ELF files have anything to do with grammars. ELF is just a container for binary data. How those data are interpreted is up to the loaders that load and interpret them. So Turing-completeness doesn't quite apply here as well.
All these years, I've been thinking ELF was "a Linux thing".
... are you saying that the Wii and the PS4 share the same executable format?
Elves terbuat dari apa sih ka?
Fun fact elf is used under brewmp.
Sometimes
In depth ELF:
Elf is the german word for the number eleven, which comes after zehn and before zwölf.
😆
hey, tell me more about the mafia
How can I encrypted ELF 64-bit /32-bit in my python script.. Full vedio needed
ELF does not have any encrption support.
0:08 well what is it, "executable" or "extendible"?
Yes!😄
It used to be extensible and now it’s executable and naturally between talking about it and making the thumbnail I got confused 😅
My mind just blew up at the half time, it was too much information O_O
Especially without explaining what things are. If I don't know what something does or is used for, it is pretty useless to know where in a file it is. It would have been better to make a series and tackle 1 part in each video. This video should have been split into at least 6 videos.
A reverse engineer expert for a hack group called "CODEX" pointed me to this video.
Nice, where do those cool guys gather?
60 fps presentation gigachad
How can i thia encryption, pls full vedio needed and WITH script installing full vedio.pls pls upload it
ELF does not have any encryption support.
I remember writing an elf parser in a coffee shop when I was homeless. Good times.
Are you ok now?
@@tanline6662 very ok.
Elf is Also used in PS2
Quite extensive.