This is an amazing resource, it's explained very carefully and considers beginners too. Thank you for taking the time to explain and share this, it's hard to find good resources to explain more complex computer science topics.
When you compile source for a specific distro how are the names and location of the dependencies for that distro determined? What tool is responsible for supplying this information? Are you supposed to manually input the paths to the dependencies when compiling? Is that what you're talking about at 5:51? I'm trying to understand why binaries might not work between distros. When you compile, you have to compile for a specific distro, right? Not just like Ubuntu, but the exact version of Ubuntu, like Ubuntu 20.04. Right? And if there are dependencies not in standard directories for the distros you are compiling for then you need to manually input dir(s) to search in for those dependencies? Is this how it works?
There are a lot of questions in here, but this is a good topic so I will do my best to answer them all. Let me answer them a little out of order. Also, everything in this response assumes we are linking dynamically and not statically and also that we are talking specifically about Linux. *I'm trying to understand why binaries might not work between distros. When you compile, you have to compile for a specific distro, right? Not just like Ubuntu, but the exact version of Ubuntu, like Ubuntu 20.04. Right?* Not really. The most important thing is that you are compiling for the correct architecture (x86, ARM, PowerPC, etc). Other than that, it might be helpful to think of it as compiling for specific libraries rather than a specific distro. When you compile an executable program using specific shared libraries, then those shared libraries need to exist on the target system. For most libraries, these do not change much from distro to distro, or at least not in a way that presents problems for linking and loading. They are generally backwards-compatible. Linux uses an interesting method called a soname to simplify shared library version changes. I explain this in one of my videos: th-cam.com/video/mUbWcxSb4fw/w-d-xo.html In a nutshell, this allows the executable program to be compiled with a generic library name that the target system loader will resolve to a specific library version and then load that version into memory. Linux shared libraries generally are very good about maintaining backwards-compatibility. This means that if you were to call a library function using version 2.20, then you would call it the same way in version 2.23 or higher. So if a program is compiled using an older version of a library, it should still work the same way (i.e. be called in the same way and return the same thing) with the newer version. However, if version 2.23 makes available a new function that was not available in 2.20 and if you compile a program that uses that new function, then if it were to run on a system using 2.20, it would not work. As a developer, you would need to be aware of this and how it would impact using your program on target systems. So, as far as compiling for distros, you really just need to focus on making sure that the libraries that you want to use are supported on that distro. For the standard libraries (libc, libm, libpthread, etc), this should rarely pose a problem. worst-case scenario, you could always compile it statically. It would have a much larger file size, and take up more memory, but then should be able to run completely on its own. One other thing to consider is the Linux kernel. If you are doing any development that requires the Linux kernel headers, then you want to make sure that you are using kernel headers for the same Linux kernel that your target system is using. This might be the case if you are creating a cross-compiler or if your project interfaces with the Linux kernel in any way. If your target system is using an older version of the Linux kernel and if your project uses kernel headers from a much newer kernel version, then this could cause some problems. *When you compile source for a specific distro how are the names and location of the dependencies for that distro determined? What tool is responsible for supplying this information?* When we talk about dependencies, there are basically two types: compile-time and run-time dependencies. Compile-time dependencies you need when you compile your program and run-time dependencies are needed on the target system to execute the program. For compile-time dependencies, the linker is responsible for locating these dependencies. As a developer, you would know the name of the library that you need to link into the program. The linker will search a set of standard directories looking for the header files (.h) and the actual libraries (.so or .a). If you are using the standard c library (libc), then you usually don't need to specify anything on the command line for the compiler (gcc). But with all other libraries, you will need to specify the name of the library that needs to be linked in with the lowercase l option (-l). For example, for libmath, the shared library is called libm and you would link it using the option -lm. With pthreads, the library is libpthread and you would link it using -lpthread. The linker then knows it needs to link in those libraries. When the compiling and linking is all done, the resultant executable program will contain the names of all of the shared libraries it needs. This is where the run-time dependencies come into play. The libraries that were linked into the program at compile-time must be available on the target system in order for it to be loaded and execute. On the target system, when the program is loaded into memory, it is the dynamic loader that is responsible for locating the shared libraries that the program needs. The loader will search a standard set of directories for the library name specified in the executable binary. When it finds the libraries, it will load them into memory if necessary, and map them into the address space of the process. *Are you supposed to manually input the paths to the dependencies when compiling? Is that what you're talking about at **5:51**?* Yes, this is just for files need for compiling. You only need to use the -I and -L options if your header files and libraries are located in a non-standard location. This is generally just the case if you are using libraries that you developed or third-party libraries that you have acquired and are linking into. If you are linking into libc, libm, libpthread, etc, then you do not need to use -I and -L because they are in standard directories and the linker will find them. However, you do still need to use the lowercase l option (-l) to let the linker know that you need those libraries to be linked in. *And if there are dependencies not in standard directories for the distros you are compiling for then you need to manually input dir(s) to search in for those dependencies? Is this how it works?* This is a good question. If you compile your program using a shared library, then that library must exist on the target system. If that library is not in a standard directory on the target system, the loader will not know how to find it. It is normally desirable to install shared libraries on the target system to one of the standard directories. However, if you do not have control over this, then you would need to know exactly which directories the libraries might be in on the target system and specify this using linker options rpath or runpath. These are options that are passed specifically to the linker at compile time and are compiled into the program executable so that the loader on the target system can locate the shared libraries. The loader will first look in runpath, then the standard directories, then rpath. Look at the official man pages for ld for more information on these options. You can pass options to the linker when using gcc with -Wl,option. See the man page for gcc for more info.
Sir, is linking necessary even if we write this code: void main(){ } If no, then can we run directly relocatable object file? If yes, then what linker does in it's linking process? By the way, very much thanks for great info!
Relocatable objects are not executable. They do not contain the information needed by the operating system to load them into memory and start them. They contain object code that is supposed to be relocated into another binary. This is what the linker does. Even if you compile the code that you provided, it would still call the linker to compile the final binary. One of the things the linker does is create the final binary, which contains all the information the loader needs to load the program into memory and start it. I plan to make videos in the future that go more into detail about the structure of the final binaries, how they are loaded and started, etc.
Technically the compiler could do that. I think that some windows compilers do it that way. I think one of the primary reasons has to do with linking rather than compiling. It's easier for the linker to work with assembly that has symbolic references to functions and data and do relocation and linking when linking the final binary.
If you compile using -g option, then would it still fold the constant and avoid using the libm shared library? Just asking, because I think -g disables many compiler optimizations, that's why 😅
I'm not sure I understand your question. I don't know what you mean by "fold the constant". The -g option puts debugging information into the ELF, but does not change how the binary is linked. If your program uses an external library, you'll still have to link using the -l option. Does that answer your question?
Wow! This series is increadible! Thank you so much, you are a life savior!
Thank you. I'm glad you find the videos useful.
You are literally a saviour..
Great tutorial, explained everything that I was confused about. Thank you
This is an amazing resource, it's explained very carefully and considers beginners too.
Thank you for taking the time to explain and share this, it's hard to find good resources to explain more complex computer science topics.
This video is exactly what I've been looking for!
Awesome job with the video. Thank you for your time and effort, this helped me out a lot as a newer SW engineer :)
Thank you so much for these videos really appreciate your contents 😀
best videos on these topics! Thanks a lot!
Thank you, excellent!!! precise, to the point tutorial and simple demonstration to understand
Thanks. I'm glad you liked it.
this is so well done, thanks!!
Sir, thank you so much for your video.
Excellent stuff !!!
Thanks!
Thans for this video!
Great video!Thanks sharing!
I love it
It is what I expected
Thank you very much
Very good tutorial. Keep it up
wow very much helpful brother!!!
Very Useful content
Good video
Sweet
Nice informative video!
Amazing video!
This video is so cool. Seriously.
For exploring the object file, you could have used objdump
Thanks sir....❤️
Great job done, good practical demo
*Please tell how to make smallest possible executables in C++.*
👍
спасибо
When you compile source for a specific distro how are the names and location of the dependencies for that distro determined? What tool is responsible for supplying this information? Are you supposed to manually input the paths to the dependencies when compiling? Is that what you're talking about at 5:51? I'm trying to understand why binaries might not work between distros. When you compile, you have to compile for a specific distro, right? Not just like Ubuntu, but the exact version of Ubuntu, like Ubuntu 20.04. Right? And if there are dependencies not in standard directories for the distros you are compiling for then you need to manually input dir(s) to search in for those dependencies? Is this how it works?
There are a lot of questions in here, but this is a good topic so I will do my best to answer them all. Let me answer them a little out of order. Also, everything in this response assumes we are linking dynamically and not statically and also that we are talking specifically about Linux.
*I'm trying to understand why binaries might not work between distros. When you compile, you have to compile for a specific distro, right? Not just like Ubuntu, but the exact version of Ubuntu, like Ubuntu 20.04. Right?*
Not really. The most important thing is that you are compiling for the correct architecture (x86, ARM, PowerPC, etc). Other than that, it might be helpful to think of it as compiling for specific libraries rather than a specific distro. When you compile an executable program using specific shared libraries, then those shared libraries need to exist on the target system. For most libraries, these do not change much from distro to distro, or at least not in a way that presents problems for linking and loading. They are generally backwards-compatible.
Linux uses an interesting method called a soname to simplify shared library version changes. I explain this in one of my videos: th-cam.com/video/mUbWcxSb4fw/w-d-xo.html
In a nutshell, this allows the executable program to be compiled with a generic library name that the target system loader will resolve to a specific library version and then load that version into memory.
Linux shared libraries generally are very good about maintaining backwards-compatibility. This means that if you were to call a library function using version 2.20, then you would call it the same way in version 2.23 or higher. So if a program is compiled using an older version of a library, it should still work the same way (i.e. be called in the same way and return the same thing) with the newer version.
However, if version 2.23 makes available a new function that was not available in 2.20 and if you compile a program that uses that new function, then if it were to run on a system using 2.20, it would not work. As a developer, you would need to be aware of this and how it would impact using your program on target systems.
So, as far as compiling for distros, you really just need to focus on making sure that the libraries that you want to use are supported on that distro. For the standard libraries (libc, libm, libpthread, etc), this should rarely pose a problem. worst-case scenario, you could always compile it statically. It would have a much larger file size, and take up more memory, but then should be able to run completely on its own.
One other thing to consider is the Linux kernel. If you are doing any development that requires the Linux kernel headers, then you want to make sure that you are using kernel headers for the same Linux kernel that your target system is using. This might be the case if you are creating a cross-compiler or if your project interfaces with the Linux kernel in any way. If your target system is using an older version of the Linux kernel and if your project uses kernel headers from a much newer kernel version, then this could cause some problems.
*When you compile source for a specific distro how are the names and location of the dependencies for that distro determined? What tool is responsible for supplying this information?*
When we talk about dependencies, there are basically two types: compile-time and run-time dependencies. Compile-time dependencies you need when you compile your program and run-time dependencies are needed on the target system to execute the program. For compile-time dependencies, the linker is responsible for locating these dependencies. As a developer, you would know the name of the library that you need to link into the program. The linker will search a set of standard directories looking for the header files (.h) and the actual libraries (.so or .a). If you are using the standard c library (libc), then you usually don't need to specify anything on the command line for the compiler (gcc). But with all other libraries, you will need to specify the name of the library that needs to be linked in with the lowercase l option (-l). For example, for libmath, the shared library is called libm and you would link it using the option -lm. With pthreads, the library is libpthread and you would link it using -lpthread. The linker then knows it needs to link in those libraries.
When the compiling and linking is all done, the resultant executable program will contain the names of all of the shared libraries it needs. This is where the run-time dependencies come into play. The libraries that were linked into the program at compile-time must be available on the target system in order for it to be loaded and execute. On the target system, when the program is loaded into memory, it is the dynamic loader that is responsible for locating the shared libraries that the program needs. The loader will search a standard set of directories for the library name specified in the executable binary. When it finds the libraries, it will load them into memory if necessary, and map them into the address space of the process.
*Are you supposed to manually input the paths to the dependencies when compiling? Is that what you're talking about at **5:51**?*
Yes, this is just for files need for compiling. You only need to use the -I and -L options if your header files and libraries are located in a non-standard location. This is generally just the case if you are using libraries that you developed or third-party libraries that you have acquired and are linking into. If you are linking into libc, libm, libpthread, etc, then you do not need to use -I and -L because they are in standard directories and the linker will find them. However, you do still need to use the lowercase l option (-l) to let the linker know that you need those libraries to be linked in.
*And if there are dependencies not in standard directories for the distros you are compiling for then you need to manually input dir(s) to search in for those dependencies? Is this how it works?*
This is a good question. If you compile your program using a shared library, then that library must exist on the target system. If that library is not in a standard directory on the target system, the loader will not know how to find it. It is normally desirable to install shared libraries on the target system to one of the standard directories. However, if you do not have control over this, then you would need to know exactly which directories the libraries might be in on the target system and specify this using linker options rpath or runpath. These are options that are passed specifically to the linker at compile time and are compiled into the program executable so that the loader on the target system can locate the shared libraries. The loader will first look in runpath, then the standard directories, then rpath. Look at the official man pages for ld for more information on these options. You can pass options to the linker when using gcc with -Wl,option. See the man page for gcc for more info.
@@embeddedarmdev really helpful explanation, probably the only time I've read a youtube comment so attentively. Thanks a bunch!
Sir, is linking necessary even if we write this code:
void main(){
}
If no, then can we run directly relocatable object file?
If yes, then what linker does in it's linking process?
By the way, very much thanks for great info!
Relocatable objects are not executable. They do not contain the information needed by the operating system to load them into memory and start them. They contain object code that is supposed to be relocated into another binary. This is what the linker does. Even if you compile the code that you provided, it would still call the linker to compile the final binary. One of the things the linker does is create the final binary, which contains all the information the loader needs to load the program into memory and start it. I plan to make videos in the future that go more into detail about the structure of the final binaries, how they are loaded and started, etc.
@@embeddedarmdev thanks for the great answer
@@embeddedarmdev you go very deep man
what is the need for an assembler why can't the compiler directly convert to machine code
Technically the compiler could do that. I think that some windows compilers do it that way. I think one of the primary reasons has to do with linking rather than compiling. It's easier for the linker to work with assembly that has symbolic references to functions and data and do relocation and linking when linking the final binary.
If you compile using -g option, then would it still fold the constant and avoid using the libm shared library? Just asking, because I think -g disables many compiler optimizations, that's why 😅
I'm not sure I understand your question. I don't know what you mean by "fold the constant". The -g option puts debugging information into the ELF, but does not change how the binary is linked. If your program uses an external library, you'll still have to link using the -l option. Does that answer your question?
i paid 20 lakhs in clg and yet my prof couldn't match the level of clarity provided here.
🤔