Maybe it's a little pedantic of me, but I'd just let Ruby read it in as binary and in my compare do "\xFF\xD8\xFF".encode('ASCII-8BIT'). We make the test bytes match what we expect rather than mutating what we're reading in to see if the mutated value matches something else. And I think there's an alias of 'BINARY' so "\xFF\xD8\xFF".encode('BINARY') should also work and is maybe more explicit about what we want.
@@Raugturi can't ruby just read the bytes as a number array and let you create a number array with 3 hex constants and compare them? Would be weird if any language couldn't do numbers amd arrays.
Thank you for the example! Really a good intro to magic bytes 🙂 Maybe another nice video would be a simple jfif/exif low-level parser - just the big stuff - display the block type and size, could be useful for integrity check 🙂 most out-of-the-box libraries “fix” minor errors or ignore erroneous information when possible.
Thanks. Good point. I've definitely still seen the bytes object cause some newcomer confusion when the programmer doesn't understanding why a bytes object and a string object (with a string of bytes) are not the same thing. It feels like a different flavor of the same problem - but maybe/probably a more sensible solution.
❤ This is one of my favorites! A real world example. Unix/Linux has "magic" built in, and I've edited /etc/magic by hand to add file types, but I do data migrations, so this indepth how-to is very helpful for building custom tools myself regardless of platform. It maybe a bit off the channel topic, but it might be useful for you to do a video on encoding - ASCII, UTF, ISO, EBCDIC - and maybe even add byte transmission like 7bit even parity, 8bit no parity...stuff like that. Working at the atomic level of data is very helpful to develop a better understanding of computers in general.
Great video as usual, What is the proper way to exit a program?..."exit(EXIT_FAILURE)" or with "return EXIT_FAILURE"...or use EXIT_SUCCESS...not sure when to use "return" or "exit" to end the program.
With most C runtimes, main is called like this from some other libC startup routine - result = main(argc, argv); exit(result); So, returning from main and calling exit are essentially doing the same thing. I suppose that calling exit from main rather than returning might be slightly more efficient (avoiding a function call return), but it's not likely to make any difference (especially once compiler optimizations get involved).
I'm on a roll here binge watching your videos. Super impressed. Hoping you can help me connect the dots of my limited understanding. I was under the impression C++ has better string handling capabilities and figured C++ would be your go-to. In this case, you chose C for it's simplicity for its lack of encoding confusion. Does that mean C++ adds complexity compared to C, such that you'd need to concern yourself with encoding choices?
Thanks! I'm glad they're helping out. C++ doesn't do any automatic string encoding stuff, and you could do essentially the same thing that I did here using C++ (fopen, fread are available in both). But, yes, C++ does provide some nice string-related tools (nowhere near what you get from python or ruby), but when I'm working with binary data and individual bytes, I often don't see an advantage to using C++, unless I need object oriented stuff elsewhere in the code. In this case, I didn't.
That's completely bonkers to me that Ruby would have that issue. Especially considering that 7-bit ASCII maps exactly into UTF-8. At this point I should probably be keeping a list of reasons to never learn Ruby. As an aside, being a Linux user I would never bother to write such a program because `file` exists and I could just check with it, however, a good example you might want to make a future video for is serializing data structures. I prefer to use a text based method, such as TOML, for simple structures, but when it's complex I use the binary approach.
I wrote an image grouping Tool that groups png files into 1 file (png hat it's own set of magic numbers) and I thought it would be problematic if my seperator identifiers would randomly appear in the binary file (unlikely but possibly). My theory (not confirmed) was that the magic numbers of the file format likely won't appear in the binary. So I literally packed the images byte to byte after each other and seperated them by splitting them at the magic start. It works and it's as efficient as possible.
Not sure I understand the question. Are you asking if the magic numbers could check out and it not follow the JPG format - maybe holding other data? Yes, it could.
@JacobSorber oh wow,.hi Jacob! I suppose yes, I'm asking if those magic numbers could pose as a jpg yet maybe carry some hidden data within those very magic numbers. I want to thank you for your reply, and will take it on board when checking an example of this case out. Cheers!
feeling like home when somebody types things in C. Things are exactly what they are in 99%, and 1% stays just for rare machines’ unique memory conventions, and nothing else.
10:17 I know it’s not super important, but I think it would make more sense to have this print “No!” instead of return an error. Because if it’s reading less, i.e. if the file is smaller than 3 bytes, then we know that it’s not a JPG. I’m not sure if it’s possible though for fread to fail, but the file may still be a JPG? Would it be safe to assume this doesn’t happen if fread does not return 3?
You can check rather a file io error occurred or you hit end of file by calling and checking feof() //end of file And ferror() // error So the number returned really just serves to confirm the success case. To fully answer your question, you could do some retrying by rerunning fread or completely reopening the file in case fread did not fail with eof (which would confirm it not being a jpg as it is smaller than 3 bytes). But I'm not sure rather putting this much effort would be worth it in most cases.
Because I usually do a typedef char byte or more specifically if I know I'm only targeting C99 and later typedef uint8_t byte just to be explicit when writing code. When I saw a char array of size 3 my brain went "that's not big enough, what about the null terminator" took my brain a second to go, "no it's just a byte array of fixed length so we know how long it is" 😅 Also only using printf for error messages instead of fprintf(stderr, ...); bugged me way more then it should lol. Good explanation though and I love the ruby example because I've run into encoding issues before with scripting languages, never had an encoding problem with C. So it's certainly something to keep in mind.
Wait until you try to use windows and c/c++ and you get char signed char unsigned char wchar_t char8_t char16_t char32_t All being different character types used in various functions in windows. Ps: And yes I looked it up char is defined as being neither unsigned char nor signed char in the Microsoft compiler.
At 16:50 isn't it better to encore the UTF-8 string into 8-bit ASCII instead? How can you be sure that the 3 bytes read from the files form valid utf-8?
Yeah, probably. I just picked one, since I just needed the encodings to match. But, yes, if I were doing anything else with the strings, forcing both to ASCII-8 would have been better.
@@JacobSorber In this particular case, since you know that the string you're looking for is not a valid UTF-8 string, I think you'd definitely be better off using the ASCII-8BIT encoding. Otherwise, you're depending on the language to do the right thing when comparing invalid UTF-8 strings. If it is doing a byte-by-byte comparison you're safe, but if "equal" means "represents the same sequence of Unicode code points", I would think all bets are off if the strings being compared are not valid UTF-8 strings. (It is possible for two UTF-8 strings that are not identical byte-wise to be equal in terms of what they encode.)
Why not #define the MAGIC_NUM_BYTES? I know the low level difference between defines and declarations, but what is the high level practical difference?
That would work, as well. I have a video about this somewhere in the list. Making it a variable allows the compiler to help you with type checking and some forms of error detection. #define might in some cases have performance advantages (I don't think it would in this case). For this example, both are viable options.
Yes, this encoding bs really frustates me when I'm doing a byte or bit level manipulation in python. Things that seemingly so easy in C can get tangled due to mismatching data type.
what IDE is that? looks like its using some syntactical sugar coating. it uses enequal symbol instead of != . small thing but kind a annoying it hides the real code hehe.
You should do proper jpg chunks reading. Testing the first 3 bytes is not enough. People can learn more about formats, and common practices how to work with binary data and maybe how to save and load their own binary files
@@RobertFletcherOBE Yep i know that. But is better to close/free all. For a short living program you dont have to use free. For programs that run always you must you use free.
I don't think reading 3 random bytes as UTF-8 is a good idea, not all 3 bytes sequences are valid UTF-8 so your program might crash, give error or return a garbage string. I think it's better to convert the string you are comparing with to ASCII / bytes / char / whatever it's called in your language
The Programmer word gave very scary Word like , CPU DPU GPU JPEG ZIP MPI LLVM …; that take me out fare but when dive in deep to understand i found it only to make different between principles; that remember me to very scared mathematical theorems that push people away to studying mathematics.
The read method in ruby has an option for encoding, e.g. f.read(3, encoding="UTF-8")
Good point. Thanks!
Maybe it's a little pedantic of me, but I'd just let Ruby read it in as binary and in my compare do "\xFF\xD8\xFF".encode('ASCII-8BIT'). We make the test bytes match what we expect rather than mutating what we're reading in to see if the mutated value matches something else. And I think there's an alias of 'BINARY' so "\xFF\xD8\xFF".encode('BINARY') should also work and is maybe more explicit about what we want.
@@Raugturi can't ruby just read the bytes as a number array and let you create a number array with 3 hex constants and compare them?
Would be weird if any language couldn't do numbers amd arrays.
@@Raugturiwas about to write the exact same comment😅
Definitely would be interested if more file format videos are to come :)
Lets have a deep dive into the meta data
Interesting. I’m taking a python course currently. This will make an interesting exercise in python. Thanks! 😀
I'm pretty sure that this is not an issue in Python though as long as binary mode is being used.
Thank you for the example! Really a good intro to magic bytes 🙂 Maybe another nice video would be a simple jfif/exif low-level parser - just the big stuff - display the block type and size, could be useful for integrity check 🙂 most out-of-the-box libraries “fix” minor errors or ignore erroneous information when possible.
Python3 doesn't have this problem: as you can read directly binary and get "bytes" objects.
Thanks. Good point. I've definitely still seen the bytes object cause some newcomer confusion when the programmer doesn't understanding why a bytes object and a string object (with a string of bytes) are not the same thing. It feels like a different flavor of the same problem - but maybe/probably a more sensible solution.
a video on reading/writing bitmap data would be cool :)
❤ This is one of my favorites! A real world example. Unix/Linux has "magic" built in, and I've edited /etc/magic by hand to add file types, but I do data migrations, so this indepth how-to is very helpful for building custom tools myself regardless of platform. It maybe a bit off the channel topic, but it might be useful for you to do a video on encoding - ASCII, UTF, ISO, EBCDIC - and maybe even add byte transmission like 7bit even parity, 8bit no parity...stuff like that. Working at the atomic level of data is very helpful to develop a better understanding of computers in general.
Yes for metadata and more file disassembly please!
I love your vids... they helped me understand pointers :)
Yessss! Glad they helped.
I would love more videos on this topic!
I would love to see some videos on binary file manipulation, especially something like writing your own encryption program.
Great video as usual, What is the proper way to exit a program?..."exit(EXIT_FAILURE)" or with "return EXIT_FAILURE"...or use EXIT_SUCCESS...not sure when to use "return" or "exit" to end the program.
With most C runtimes, main is called like this from some other libC startup routine - result = main(argc, argv); exit(result); So, returning from main and calling exit are essentially doing the same thing. I suppose that calling exit from main rather than returning might be slightly more efficient (avoiding a function call return), but it's not likely to make any difference (especially once compiler optimizations get involved).
Thanks Jacob, great video, lets play with sockets and threads, I think it will be more fun.
I'm on a roll here binge watching your videos. Super impressed. Hoping you can help me connect the dots of my limited understanding. I was under the impression C++ has better string handling capabilities and figured C++ would be your go-to. In this case, you chose C for it's simplicity for its lack of encoding confusion. Does that mean C++ adds complexity compared to C, such that you'd need to concern yourself with encoding choices?
Thanks! I'm glad they're helping out. C++ doesn't do any automatic string encoding stuff, and you could do essentially the same thing that I did here using C++ (fopen, fread are available in both). But, yes, C++ does provide some nice string-related tools (nowhere near what you get from python or ruby), but when I'm working with binary data and individual bytes, I often don't see an advantage to using C++, unless I need object oriented stuff elsewhere in the code. In this case, I didn't.
That's completely bonkers to me that Ruby would have that issue. Especially considering that 7-bit ASCII maps exactly into UTF-8. At this point I should probably be keeping a list of reasons to never learn Ruby. As an aside, being a Linux user I would never bother to write such a program because `file` exists and I could just check with it, however, a good example you might want to make a future video for is serializing data structures. I prefer to use a text based method, such as TOML, for simple structures, but when it's complex I use the binary approach.
I wrote an image grouping Tool that groups png files into 1 file (png hat it's own set of magic numbers) and I thought it would be problematic if my seperator identifiers would randomly appear in the binary file (unlikely but possibly). My theory (not confirmed) was that the magic numbers of the file format likely won't appear in the binary.
So I literally packed the images byte to byte after each other and seperated them by splitting them at the magic start. It works and it's as efficient as possible.
Could you check if it is not a real jpg but may have some hidden data within the file?
I know probably not to expect any reply from Jacob 😢
Not sure I understand the question. Are you asking if the magic numbers could check out and it not follow the JPG format - maybe holding other data? Yes, it could.
@JacobSorber oh wow,.hi Jacob!
I suppose yes, I'm asking if those magic numbers could pose as a jpg yet maybe carry some hidden data within those very magic numbers.
I want to thank you for your reply, and will take it on board when checking an example of this case out.
Cheers!
Why does the c version work? FF and D8 are outside the range of a char, which is normally signed.
Shouldn't the compiler warn about it?
feeling like home when somebody types things in C. Things are exactly what they are in 99%, and 1% stays just for rare machines’ unique memory conventions, and nothing else.
Helpful presentation!
Why put env in shebang line instead of interpreter alone?
Seems would be prettier with collapsed or squashed explorer.
That's pretty cool. I've never tried to do this before.
I love this way of questioning everything. Next: How to tell if a JPEG is a file.
You would have to define what you mean by "JPEG". 🤔
@@JacobSorber that is easy, just #define JPEG
6:51 nice jab at Windows
10:17 I know it’s not super important, but I think it would make more sense to have this print “No!” instead of return an error.
Because if it’s reading less, i.e. if the file is smaller than 3 bytes, then we know that it’s not a JPG.
I’m not sure if it’s possible though for fread to fail, but the file may still be a JPG? Would it be safe to assume this doesn’t happen if fread does not return 3?
You can check rather a file io error occurred or you hit end of file by calling and checking
feof() //end of file
And
ferror() // error
So the number returned really just serves to confirm the success case.
To fully answer your question, you could do some retrying by rerunning fread or completely reopening the file in case fread did not fail with eof (which would confirm it not being a jpg as it is smaller than 3 bytes).
But I'm not sure rather putting this much effort would be worth it in most cases.
Because I usually do a
typedef char byte
or more specifically if I know I'm only targeting C99 and later
typedef uint8_t byte
just to be explicit when writing code. When I saw a char array of size 3 my brain went "that's not big enough, what about the null terminator" took my brain a second to go, "no it's just a byte array of fixed length so we know how long it is" 😅
Also only using printf for error messages instead of fprintf(stderr, ...); bugged me way more then it should lol. Good explanation though and I love the ruby example because I've run into encoding issues before with scripting languages, never had an encoding problem with C. So it's certainly something to keep in mind.
Wait until you try to use windows and c/c++ and you get
char
signed char
unsigned char
wchar_t
char8_t
char16_t
char32_t
All being different character types used in various functions in windows.
Ps:
And yes I looked it up char is defined as being neither unsigned char nor signed char in the Microsoft compiler.
@@redcrafterlppa303 Yeah that's why I use linux 🤣
Just found out about this channel. Nice content!
Thanks, and welcome!
At 16:50 isn't it better to encore the UTF-8 string into 8-bit ASCII instead?
How can you be sure that the 3 bytes read from the files form valid utf-8?
Yeah, probably. I just picked one, since I just needed the encodings to match. But, yes, if I were doing anything else with the strings, forcing both to ASCII-8 would have been better.
@@JacobSorber In this particular case, since you know that the string you're looking for is not a valid UTF-8 string, I think you'd definitely be better off using the ASCII-8BIT encoding. Otherwise, you're depending on the language to do the right thing when comparing invalid UTF-8 strings. If it is doing a byte-by-byte comparison you're safe, but if "equal" means "represents the same sequence of Unicode code points", I would think all bets are off if the strings being compared are not valid UTF-8 strings. (It is possible for two UTF-8 strings that are not identical byte-wise to be equal in terms of what they encode.)
Why not #define the MAGIC_NUM_BYTES? I know the low level difference between defines and declarations, but what is the high level practical difference?
That would work, as well. I have a video about this somewhere in the list. Making it a variable allows the compiler to help you with type checking and some forms of error detection. #define might in some cases have performance advantages (I don't think it would in this case). For this example, both are viable options.
"Strings are strings; bytes are bytes." That's the way it should always be! Good video.
How are you using that not equal sign?
Please make a next video of socket server handle part 4 - event driven
Any reason as to why you never closed the FILE * ?
Yes, this encoding bs really frustates me when I'm doing a byte or bit level manipulation in python. Things that seemingly so easy in C can get tangled due to mismatching data type.
The worst thing is you don't directly see the datatypes because python is stupid (sorry not sorry)
Great video!
basically an oddly specific version of "file" command/program
what IDE is that? looks like its using some syntactical sugar coating. it uses enequal symbol instead of != . small thing but kind a annoying it hides the real code hehe.
You should do proper jpg chunks reading. Testing the first 3 bytes is not enough. People can learn more about formats, and common practices how to work with binary data and maybe how to save and load their own binary files
wide char argv tutorial for Win32 wen (wmain and wWinMain)?
Nice!
Sir, Suggest me a book where I can learn about these things...
Greetings from San Francisco. Let's get to the important stuff: Where can I get one of those malloc() T-shirts? It is not on you Merch section.
I'm glad you have your priorities straight. It should be on there now.
@@JacobSorber I'll get my order in now, thanks.
did you forgot to close file?
Yeo i see no fclose(fp);
when a process exits its resources are released.
@@RobertFletcherOBE Yep i know that.
But is better to close/free all.
For a short living program you dont have to use free. For programs that run always you must you use free.
Yes, I did. Sorry about that.
@@RobertFletcherOBE ok but it would be nice to manually release resource when it's not needed any more
I don't think reading 3 random bytes as UTF-8 is a good idea, not all 3 bytes sequences are valid UTF-8 so your program might crash, give error or return a garbage string. I think it's better to convert the string you are comparing with to ASCII / bytes / char / whatever it's called in your language
The Programmer word gave very scary Word like , CPU DPU GPU JPEG ZIP MPI LLVM …; that take me out fare but when dive in deep to understand i found it only to make different between principles; that remember me to very scared mathematical theorems that push people away to studying mathematics.
Pretty nicee
For those using GCC, you have to initialize magicNumber using malloc:
char *magicNumber = malloc(MAGIC_NUM_BYTES * sizeof(char));