I must have watched half a dozen videos about encoding text over the years, but this one is easily the best one! Very easy to understand with all the examples. Now to hope I don't ever have to deal with anything else than utf-8...
Probably the clearest explanation I've seen on the topic. There's lots that cover the basics pretty well, but completely skip over the byte order aspect.
This was super interesting and well done, thank you so much! I had an issue with BOM yesterday that broke my script because the shebang wasn't being recognized. I love discovering Hex Editor Neo, gave me a lot of peace of mind, actually seeing the "ef bb ef" in front of my code haha.
As a Japanese speaker, the fact that the first four glyphs in your thumbnail are "I❤日本" made me wonder if you're a fan of Japan! (日本 means Japan in Japanese)
Quite excellent. So many little places it is possible, even easy to trip up and give sub-par or confusing information on this topic, and you just tap dance right across that minefield. Beautiful.
One question at the end. There are 17 * 256 * 256 possible characters across the 17 planes, not including the various blackout or restricted sections like those used for surrogates...isn't that more like: 1,114,111 than well over 2 million? They do require 21 bits, rather than 20, because they left all but 2048 of the original BMP to lie where they were, and added another 1024 * 1024 possible new ones on top of that, so it is (256 * 256) * 16 + (256 * 256) - 2048...either way, way more than we should ever need if we don't go nuts...as they point out, Unicode is for characters, not glyphs or fonts, so if there are 500 ways you want to write A, that is still just one code point...
To be honest, I think Unicode is one of those topics which, if you keep digging, you may well go nuts. I made a conscious decision not to overthink it (i.e. "tap dance"). Perhaps Unicode will prove inadequate if SETI ever come up with the goods. :)KD
@@ComputerScienceLessons Yeah, one thing that is unfortunate is that we went from a situation where almost everything "normally" used was in BMP, to one where lots of 🤦♂️🤷♂️🤞super-popular stuff ain't. Too bad!
Great lecture! I had a question: what is that weird characters we see on the Hex editor? Because Notepad recognizes the encoding used, while Hex editor doesn't.
Hex editor displays the raw data present on the file. It could have shown that to us in binary format too, but it displays that raw binary data to us in a simpler Hexadecimal format
what web site do i get these things from? you talk about it 16:00 but were are they? how do i find them? what do i search for? this would make a good video.
It was brought to my attention by one of my viewers that the bit sequence for the Greek letter Phi, in the table of UTF-8 values, did not match the bit sequence that was (correctly) derived below the table (thank you!). All of the other information was correct. I fixed the problem and took the opportunity to de-ess the sound at the same time :)KD
Dear Sir, I have tried to convert '€' , in pure binary It is 10000000 and in denary 128, to UTF-8 format. As It is 1 byte character but also starts with 1 . Hence I am confused how to convert it to UTF-8 format. Please help. Thanks in Advance.
It doesn't fit in 7 bits, so it's not 1 byte in UTF-8. It does however fit in 11 bytes, so it's a 2 byte codepoint. Pad to 11 bits and shove it in the two bytes with the proper headers 00010_000000 => 110_00010 10_000000
4:56 in UTF-8 we have control bits which are necessary to identify the number of bytes used to represent a code point while in UTF-16 we don't have such thing because low and high surrogates are known by the the range of values so we can identify if a code point is encoded using 1 or 2 UTF-16 code units by just the range of values can some one correct me please ?
@@justcurious1940 That doesn't look quite right... Did you mean to say 2🐝∨¬2🐝? Edit: oh, I guess that was the binary string showed in the video? Then it's possible the video did a typo or something.
And there are even more stupid encodings like Oracle's CESU-8, which mostly is UTF-8, but encodes chars outside the BMP as UTF-16 and then those surrogates *again* as UTF-8, so it becomes 6 bytes... WTF?
@@ComputerScienceLessons i could not find an online website that convert UTF- 8 data to txt directly i had to convert them to unicode code points first then convert each code point to the corresponding character 😅
I must have watched half a dozen videos about encoding text over the years, but this one is easily the best one! Very easy to understand with all the examples. Now to hope I don't ever have to deal with anything else than utf-8...
Thank you. I think UTF-8 will be around until we start communicating with extra-terrestrials. :)KD
I'm a CS graduate and I needed a refresher about this subject and this explanation had everything that I needed. Thanks.
You are most welcome :)KD
Probably the clearest explanation I've seen on the topic. There's lots that cover the basics pretty well, but completely skip over the byte order aspect.
Thank you :)KD
I have been waiting for this clear and precise explanation for 20 years I believe. Thanks so much.
Seriously underrated video. Should have a LOT more view than this.
Thank you :)KD
This was super interesting and well done, thank you so much! I had an issue with BOM yesterday that broke my script because the shebang wasn't being recognized. I love discovering Hex Editor Neo, gave me a lot of peace of mind, actually seeing the "ef bb ef" in front of my code haha.
Delighted to be of service :)KD
Watched all the videos in this playlist. Can't thank you enough.
You're most welcome. Thanks for watching :)KD
Best video on youtube about Unicode.
Thank you :)KD
As a Japanese speaker, the fact that the first four glyphs in your thumbnail are "I❤日本" made me wonder if you're a fan of Japan! (日本 means Japan in Japanese)
Konnichiwa - Japan is at the top of my places to visit, one day. :)KD
Quite excellent. So many little places it is possible, even easy to trip up and give sub-par or confusing information on this topic, and you just tap dance right across that minefield. Beautiful.
You are very kind. Thank you :)KD
another good one keep doing the great work man
23:15 that's the kind of nonchalant joke I like lmao
I couldn't resist :)KD
What is the joke? That he is using Notepad?
Thanks for the great video.
What is the default Endianness for most Windows file types? Is it always Big-Endian unless otherwise specified?
One question at the end. There are 17 * 256 * 256 possible characters across the 17 planes, not including the various blackout or restricted sections like those used for surrogates...isn't that more like: 1,114,111 than well over 2 million? They do require 21 bits, rather than 20, because they left all but 2048 of the original BMP to lie where they were, and added another 1024 * 1024 possible new ones on top of that, so it is (256 * 256) * 16 + (256 * 256) - 2048...either way, way more than we should ever need if we don't go nuts...as they point out, Unicode is for characters, not glyphs or fonts, so if there are 500 ways you want to write A, that is still just one code point...
To be honest, I think Unicode is one of those topics which, if you keep digging, you may well go nuts. I made a conscious decision not to overthink it (i.e. "tap dance"). Perhaps Unicode will prove inadequate if SETI ever come up with the goods. :)KD
@@ComputerScienceLessons Yeah, one thing that is unfortunate is that we went from a situation where almost everything "normally" used was in BMP, to one where lots of 🤦♂️🤷♂️🤞super-popular stuff ain't. Too bad!
So why is UTF16 the only standard not compatible with ASCII?
thanks a lot for the awesome explanation!
You're very welcome indeed :)KD
To be or not to be... good one! ;)
🤡 :)KD
Amazing video
Thanks a million :)KD
oh i thought UTF-16LE Order bytes will be for all 4 bytes .. but its only for 2 bytes-2bytes
Thank you man ..
You're welcome :)KD
1:19 nice
🤐
@@ComputerScienceLessons 😂 great video as always 👍🏻
Great lecture!
I had a question: what is that weird characters we see on the Hex editor? Because Notepad recognizes the encoding used, while Hex editor doesn't.
Hex editor displays the raw data present on the file. It could have shown that to us in binary format too, but it displays that raw binary data to us in a simpler Hexadecimal format
what web site do i get these things from? you talk about it 16:00 but were are they? how do i find them? what do i search for? this would make a good video.
may i know why this video got re-uploaded ; as if any you've any changes it would be useful for to spot
It was brought to my attention by one of my viewers that the bit sequence for the Greek letter Phi, in the table of UTF-8 values, did not match the bit sequence that was (correctly) derived below the table (thank you!). All of the other information was correct. I fixed the problem and took the opportunity to de-ess the sound at the same time :)KD
I am convinced that you are actually javidx9 in disguise
Nope - but I must admit - he looks a bit like me :)KD
@@ComputerScienceLessons Sounds similar as well haha
@23:18 "… the heart is broken" lol
Dear Sir,
I have tried to convert '€' , in pure binary It is 10000000 and in denary 128, to UTF-8 format. As It is 1 byte character but also starts with 1 . Hence I am confused how to convert it to UTF-8 format.
Please help.
Thanks in Advance.
It doesn't fit in 7 bits, so it's not 1 byte in UTF-8. It does however fit in 11 bytes, so it's a 2 byte codepoint. Pad to 11 bits and shove it in the two bytes with the proper headers
00010_000000 => 110_00010 10_000000
4:56 in UTF-8 we have control bits which are necessary to identify the number of bytes used to represent a code point while in UTF-16 we don't have such thing because low and high surrogates are known by the the range of values so we can identify if a code point is encoded using 1 or 2 UTF-16 code units by just the range of values can some one correct me please ?
2🐝⊕¬2🐝?
@@justcurious1940 That doesn't look quite right... Did you mean to say 2🐝∨¬2🐝?
Edit: oh, I guess that was the binary string showed in the video? Then it's possible the video did a typo or something.
@@angeldude101 did u convert it to binary unicode then to characters ?
@@justcurious1940 No, just noticed that ⊕ looked more like a direct/tensor sum than an or, so I just replaced it with the actual or symbol.
@@angeldude101 yea i see i think u have to it yourself because i couldn't find a website that does it directly
@24:05 UTF-8 encoding for Greek character Phi is 11001110 10100110 and not 11000011 10100110
That's a capital Phi in my video :)KD
@@ComputerScienceLessons I meant also capital letter. Or maybe I am missing sth. Capital Phi: U+03A6 and small letter Phi: U+03C6
And there are even more stupid encodings like Oracle's CESU-8, which mostly is UTF-8, but encodes chars outside the BMP as UTF-16 and then those surrogates *again* as UTF-8, so it becomes 6 bytes... WTF?
At one point in time, it was like the Wild West. :)KD
2🐝⊕¬2🐝?
THAT is the question! :)KD
@@ComputerScienceLessons i could not find an online website that convert UTF- 8 data to txt directly i had to convert them to unicode code points first then convert each code point to the corresponding character
😅
U+0045
E
Not simple but simply waste
?