@@energy-tunes Computerphile presentation appears cluttered to me. While Alex seems to be tidy & to the point with flow reminiscent to that of a river.
@@ankitchabarwal6814 nothing about it is "cluttered". the difference is that computerphile covers the most important aspect of utf 8: how it knows how many bytes to read for multibyte codepoints, while this one leaves it out. i am not blaming this video, as it is a little confusing to explain, but this video is by no means the best.
@@rz2374 It is best in one aspect: Keep it to the general encoding itself. Only one enhancement in aspect to UTF8: Pointing out to look for the UTF8-encoding strategy elsewhere…
Another fun fact about the way letters are laid out in ASCII: A capital letter's corresponding lowercase counterpart is exactly 32 values ahead. This is because 32 is a power of 2, and makes it so that you only need to flip the 6th bit (from the right) of a byte to change the case of a letter.
NOT ONLY ASCII I was going to say that it lost it's use in Unicode because it doesn't work with other alphabets, but went to check and… It works for Cyrillic and Greek as well! 1101000010010000 is А (cyr) 1101000010110000 is а 1100111010010001 is Α (greek) 1100111010110001 is α 1101000010010110 is Ж 1101000010110110 is ж Edit: Although it doesn't work for every character, 1101000010101111 is Я and 1101000110001111 is я
@@bororobo3805That's not true, most programmers if they're good, will bitwise AND or OR rather than doing an arithmetic operation. With a barrel shifter, which most CPUs have, it will always be more efficient than using an arithmetic operation.
Great explanation, thank you! For anyone who may still not understand, UTF-8 CAN get up to 32 bits and bit as big as UTF-32, but only if it has to. Otherwise, it just uses the minimum amount of space (8 bits) and expands as required, depending on the grapheme.
@@friedrichmyers You mean 32 bits. And it depends on your use case. What do you mean by typed systems language? If you use 32 bits at all times you will be using a lot of memory. If the programming language you are using is modern, it should be able to handle UTF-8 just fine and expand variably, as required.
One small correction: grapheme is a part of a particular writing system. Writing systems are always language-related. Unicode does not reflect any particular writing system. Unicode, and this is probably the smartest choice that could ever be made, maps numbers (code points) to character descriptions or names. This way Unicode is detached from any font and therefore from any particular shapes. This results in a code point relating to not what we want to see, but what we want, making it more abstract. A good examples are U+0067 and U+0261. In most writing systems based on Latin script, they are allographs, variants of grapheme, so if Unicode were to contain graphemes, there should be only one character. But this is not the case. A writing system may prefer particular glyph (a shape of a letter) and that glyph could be the main variant (allograph) of the grapheme in that system. In most Latin-based writing systems is the main variant, but in the International Phonetic Alphabet it is , because of it correlation with other similar characters like .
I took a long searching to understand the Unicode (also UTF-8 and the others), I did, but some things were still ambiguous to me, this guy literally taught me the whole topic with only 10 minutes
You earned a subscriber today. I've worked in the past with non-english characters scratching my head. This alone sums up all the concepts in details. Very good use of examples and video production 💌
7:50 Worth pointing out that Python 2 is way past its end of life now and in Python 3, all strings are Unicode aware and the u modifier does nothing (it's only there for backwards compatibility). Using Python 2 is a good way to showcase the difference between Unicode aware and unaware functions but seems like this could confuse some beginners trying to replicate what you're doing who will likely be using Python 3 and might not be aware of the difference.
Only officially and even then was another point release considered. For some platforms there is no py3 port and plenty of py2 code is still around being used.
@@mokovec It's definitely still out there but by now the official end of life has already been almost 3 years ago and it's finally dying for real. The consideration of "another point release" has long passed. On my distro, python 2 isn't installed by default anymore and on Windows, you need to really go out of your way to install it. Platforms where 2 is still the only option are far and few in between and certainly not the ones beginners are using. And there's definitely no reason to encourage starting to use it now.
@@1vader this video is not a programming course, it's an explainer on utf-8. i assume he chose python2 on purpose specifically to demonstrate the differences in a simple language (so that a beginner wouldn't get too distracted by the syntax) that has an explicit delineation between unicode and non-unicode strings. of course we shouldn't be writing production ready code based off this video lol, and the video isn't encouraging it (he literally says "check out the applicable string behaviour and libs in your own language"); it's just a teaching tool
Thanks, a ton!! Here is what I learnt from the video (I have added a few things that I knew earlier): - UTF-8 (Unicode Transformation Format, 8-bit) is an encoding scheme for representing Unicode characters. - In UTF-8, ASCII characters are represented using a single byte, which means that any valid ASCII text is also valid UTF-8 text. - Therefore, UTF8 is backward compatible with ASCII. - In UTF-8, characters that can be represented using a single byte (i.e., ASCII characters) are represented as themselves. - Characters that require more than one byte are encoded using a combination of multiple bytes. - A code point refers to a numerical value assigned to each character or symbol in the Unicode standard. - Code points are represented using hexadecimal notation and are typically prefixed with "U+" to distinguish them from other numerical values. - For example, character "é" (Latin Small Letter E with Acute) consists of two Unicode code points: the base character "e" (U+0065) and the combining acute accent (U+0301). When encoded in UTF-8, "é" is represented by the bytes 0xC3 0xA9. - A grapheme refers to a visual unit of a written language. It represents a single user-perceived character or a combination of characters that are displayed together. - len() function returns the number of bytes, not the number of characters in a Unicode-unaware string. - len() function returns the number of characters in case of a Unicode-aware string.
Trying to be better in IT and dreaming to be an awesome programmer! I have always just skipped on learning Unicode and never cared due to laziness. But now realize the very importance! thank you so much for this video.
Definitely the best explanation of Unicode I could ask for. Checked with Elixir and it appears that Elixir stringս are Unicode aware by default! Amazing!
You have an amazing talent for teaching m8! Keep it up, this video like your others is so helpful. I love that you cover basic concepts of coding, not "how do I implement a server in node.js" but what really is the essence of becoming a good coder.
Traditionally major Chinese coding methods would just consider a Chinese grapheme as 2 characters though, because they would use 2 byte coding while the the 128 ASCII code points only use 1 byte coding. And also, traditionally a Chinese grapheme would always take up double the width of an ASCII grapheme in fixed width console font. This kinda makes everything neatly aligned (the amount of storage bytes needed is the same as the amount of character printing space needed), but basically falls apart when Internet and UTF-8 become more popular. And that's also basically the very reason there are double width Latin letters in Unicode. Traditionally it's used to improve readability of English words in vertical text arrangement for Chinese and Japanese, and is called full-width letters full width as in it's the full width of a Chinese character.
I've got a paper that tells exactly how you can convert code point to UTF-8 bytes and vice versa. It's actually pretty good piece of knowledge and it tells exactly how UTF-8 encodes code points. Had I known that before, I could simply convert the Unicode symbols by hand myself.
The reason I'm going to watch this one is that it explains new stuff like Emojis without expecting already having knowledge about Unicode. Bad informative material usually either lacks the new stuff or expected to much knowledge about the old.
This is such a great explainer video. As somebody who's run into the codepoint slicing problem in Python a lot. This explains so much! Thank You again!
I kinda hoped this video would teach me how to use all the unicode characters in 11min, but it turns out it's just a general overview of how UTF-8 works. This is like going to a math class and learning that addition exists, but not learning how to use the + man.
This is so awesome yet somehow has so few views. These kind of videos are amazing for people self-studying CS. I hope you have lots more videos like this to make it easier. Keep up the great work 🤘
no videos were explaining clearly the step of converting the "grapheme" to a "code point" and then to binary they were just assuming that middle step was understood. that's what I was missing apparently thanks great video
8:15 In python3, at least now it can do that with python without any problem. >>> a="你好" >>> a[0] '你' In python3, every string is in utf8. Using a u"" is a thing in py2
Technically, the internal representation of strings in Python is actually UTF-32 or UCS-4 (they're basically the same), because strings need to be efficiently indexable - as these encoding are uniform-length, that's easy to do. Using utf-8, the lookup of a given character in a string would take longer, because the whole string up to that point would need to be traversed as bytes are not evenly allocated to each character. To top things off, strings that only contain Latin characters just use ascii anyway internally to reduce waste. That's why when a string is appended to, it creates an entire new string object, as the characters the new string has to encode may be outside the original encoding's range.
Fun fact: most writing systems can express their characters in 2 o 3 UTF-8 bytes (only 2 for the most used ones: extended latin, arabic, cyrilic, greek…). The only outlier is CJK Ideographs (Chinese, Japanese and Korean). There are an insane amount of them so each CJK grapheme carries more meaning (like a word) so it makes sense that they take more bytes in UTF-8 (usually 4 bytes).
Such best videos better than 100X bad books, I can't stress too much how good this video is. There are always treasure videos like this on TH-cam, hope OP keeps working, keep letting your knowledge and legacy to influence more people.
WHY? WHY? WHY HAVENT I SEEN YOUR CHANNEL, YOU ARE THE BEST HANDS DOWN, definitely sub + bell (and like) just the way you explain.. its so good and clear
Such clean video format for explanning clearly a knowledge ! Thank you^^ (by a newcomer in the field of informaticL May you have a nice day with cookies and tea/milk 🥛🍪
Some video games use a custom encoding/decoding systems. Like Resident Evil 4. I learned that while trying to translate the game to Arabic. Interesting stuff.
Recently I was rewriting my python program which prints all the Unicode Characters (without the combining ones) within a range of Hexadecimal characters or chosen unicode blocks. You can see it on my channel but know that i am a beginner and i am aware of how i can improve it because I created it a long time ago.
this alone is the best unicode video explanation in the entire youtube, 100x better than the, maybe, second place from Computerphile.
Computerphile's explanation was very concise as well. I don't understand why you are blatantly throwing shade on them like that lol
@@energy-tunes Computerphile presentation appears cluttered to me. While Alex seems to be tidy & to the point with flow reminiscent to that of a river.
@@ankitchabarwal6814 nothing about it is "cluttered". the difference is that computerphile covers the most important aspect of utf 8: how it knows how many bytes to read for multibyte codepoints, while this one leaves it out. i am not blaming this video, as it is a little confusing to explain, but this video is by no means the best.
@@rz2374 It is best in one aspect: Keep it to the general encoding itself. Only one enhancement in aspect to UTF8: Pointing out to look for the UTF8-encoding strategy elsewhere…
i didnt understand clearly ... Can you explain me?
Another fun fact about the way letters are laid out in ASCII: A capital letter's corresponding lowercase counterpart is exactly 32 values ahead. This is because 32 is a power of 2, and makes it so that you only need to flip the 6th bit (from the right) of a byte to change the case of a letter.
ASCII digits are also nicely laid out; the low nibble of characters '0' to '9' are 0 to 9.
Yup. You only need to XOR to flip back and forth
@@Michael75579 yup. You can subtract from '0' to get decimal equivalents.
This is how most assembly programers do it
NOT ONLY ASCII
I was going to say that it lost it's use in Unicode because it doesn't work with other alphabets, but went to check and… It works for Cyrillic and Greek as well!
1101000010010000 is А (cyr)
1101000010110000 is а
1100111010010001 is Α (greek)
1100111010110001 is α
1101000010010110 is Ж
1101000010110110 is ж
Edit: Although it doesn't work for every character,
1101000010101111 is Я and
1101000110001111 is я
@@bororobo3805That's not true, most programmers if they're good, will bitwise AND or OR rather than doing an arithmetic operation. With a barrel shifter, which most CPUs have, it will always be more efficient than using an arithmetic operation.
Great explanation, thank you! For anyone who may still not understand, UTF-8 CAN get up to 32 bits and bit as big as UTF-32, but only if it has to. Otherwise, it just uses the minimum amount of space (8 bits) and expands as required, depending on the grapheme.
So I would still allocate 32 bytes, right?
I use a typed systems language.
@@friedrichmyers You mean 32 bits. And it depends on your use case. What do you mean by typed systems language? If you use 32 bits at all times you will be using a lot of memory. If the programming language you are using is modern, it should be able to handle UTF-8 just fine and expand variably, as required.
This is the type of content that should be suggested by TH-cam to everyone. Great explanation, thankyou
I've listened to quite a few Unicode tutorials. This one blows the others out of the water. Clear. Concise. Good tempo. Thx
FINALLY someone is able to explain this clearly. Most other videos complicate this so much. Thank you!
One small correction: grapheme is a part of a particular writing system. Writing systems are always language-related. Unicode does not reflect any particular writing system. Unicode, and this is probably the smartest choice that could ever be made, maps numbers (code points) to character descriptions or names. This way Unicode is detached from any font and therefore from any particular shapes. This results in a code point relating to not what we want to see, but what we want, making it more abstract.
A good examples are U+0067 and U+0261. In most writing systems based on Latin script, they are allographs, variants of grapheme, so if Unicode were to contain graphemes, there should be only one character. But this is not the case. A writing system may prefer particular glyph (a shape of a letter) and that glyph could be the main variant (allograph) of the grapheme in that system. In most Latin-based writing systems is the main variant, but in the International Phonetic Alphabet it is , because of it correlation with other similar characters like .
I took a long searching to understand the Unicode (also UTF-8 and the others), I did, but some things were still ambiguous to me, this guy literally taught me the whole topic with only 10 minutes
You earned a subscriber today.
I've worked in the past with non-english characters scratching my head.
This alone sums up all the concepts in details. Very good use of examples and video production 💌
Man this channel is golden. The most difficult topics here are explained so easily and in such a unique way
This video is like the sum of the most important things about unicode and ascii, very well done.
👍👍🏾
Wow being the messy programmer that I am I always got encoded and decoded mixed up... Much clearer now, thanks
7:50 Worth pointing out that Python 2 is way past its end of life now and in Python 3, all strings are Unicode aware and the u modifier does nothing (it's only there for backwards compatibility). Using Python 2 is a good way to showcase the difference between Unicode aware and unaware functions but seems like this could confuse some beginners trying to replicate what you're doing who will likely be using Python 3 and might not be aware of the difference.
This also caught me off guard. Python 2 has been deprecated for such a long time by now.
Only officially and even then was another point release considered. For some platforms there is no py3 port and plenty of py2 code is still around being used.
@@mokovec It's definitely still out there but by now the official end of life has already been almost 3 years ago and it's finally dying for real. The consideration of "another point release" has long passed. On my distro, python 2 isn't installed by default anymore and on Windows, you need to really go out of your way to install it. Platforms where 2 is still the only option are far and few in between and certainly not the ones beginners are using. And there's definitely no reason to encourage starting to use it now.
@@1vader I don't see anyone encouraging it.
@@1vader this video is not a programming course, it's an explainer on utf-8.
i assume he chose python2 on purpose specifically to demonstrate the differences in a simple language (so that a beginner wouldn't get too distracted by the syntax) that has an explicit delineation between unicode and non-unicode strings. of course we shouldn't be writing production ready code based off this video lol, and the video isn't encouraging it (he literally says "check out the applicable string behaviour and libs in your own language"); it's just a teaching tool
Thanks, a ton!!
Here is what I learnt from the video (I have added a few things that I knew earlier):
- UTF-8 (Unicode Transformation Format, 8-bit) is an encoding scheme for representing Unicode characters.
- In UTF-8, ASCII characters are represented using a single byte, which means that any valid ASCII text is also valid UTF-8 text.
- Therefore, UTF8 is backward compatible with ASCII.
- In UTF-8, characters that can be represented using a single byte (i.e., ASCII characters) are represented as themselves.
- Characters that require more than one byte are encoded using a combination of multiple bytes.
- A code point refers to a numerical value assigned to each character or symbol in the Unicode standard.
- Code points are represented using hexadecimal notation and are typically prefixed with "U+" to distinguish them from other numerical values.
- For example, character "é" (Latin Small Letter E with Acute) consists of two Unicode code points: the base character "e" (U+0065) and the combining acute accent (U+0301). When encoded in UTF-8, "é" is represented by the bytes 0xC3 0xA9.
- A grapheme refers to a visual unit of a written language. It represents a single user-perceived character or a combination of characters that are displayed together.
- len() function returns the number of bytes, not the number of characters in a Unicode-unaware string.
- len() function returns the number of characters in case of a Unicode-aware string.
thanks for posting the summary here
The notes on pythons functions are wrong as they are for a much earlier version that has not been in use for half a decade
I keep coming back to this video to refresh the concept, thank you :)
Trying to be better in IT and dreaming to be an awesome programmer! I have always just skipped on learning Unicode and never cared due to laziness. But now realize the very importance! thank you so much for this video.
been using Unicode for so long but never really understood what it means. Thanks for explaination.
The clearest Unicode has ever been in my career!
Definitely the best explanation of Unicode I could ask for. Checked with Elixir and it appears that Elixir stringս are Unicode aware by default! Amazing!
Your videos have helped reach over $200,000 in stocks by age 23! Thanks In The setup. Keep the videos coming.
this is such an appealing video due to the detail, music, subtle animations, even the colour theme. thanks for the video!
You have an amazing talent for teaching m8! Keep it up, this video like your others is so helpful. I love that you cover basic concepts of coding, not "how do I implement a server in node.js" but what really is the essence of becoming a good coder.
The atmosphere in the video is so chill, and the video is informative, TH-cam recommend this to more people.
I hit the (U+1F44D) button because this video was so easy to understand!
Traditionally major Chinese coding methods would just consider a Chinese grapheme as 2 characters though, because they would use 2 byte coding while the the 128 ASCII code points only use 1 byte coding. And also, traditionally a Chinese grapheme would always take up double the width of an ASCII grapheme in fixed width console font. This kinda makes everything neatly aligned (the amount of storage bytes needed is the same as the amount of character printing space needed), but basically falls apart when Internet and UTF-8 become more popular.
And that's also basically the very reason there are double width Latin letters in Unicode. Traditionally it's used to improve readability of English words in vertical text arrangement for Chinese and Japanese, and is called full-width letters full width as in it's the full width of a Chinese character.
Very useful and applicable explanation of ASCII, UTF and UNICODE
I've got a paper that tells exactly how you can convert code point to UTF-8 bytes and vice versa. It's actually pretty good piece of knowledge and it tells exactly how UTF-8 encodes code points. Had I known that before, I could simply convert the Unicode symbols by hand myself.
Where have you been all my life?! 🤩 What a great explanation! Thank you so very much! 🥰
This is the best explanation of unicode and UTF-8. Super comprehensive & helpful! Thanks!
The reason I'm going to watch this one is that it explains new stuff like Emojis without expecting already having knowledge about Unicode. Bad informative material usually either lacks the new stuff or expected to much knowledge about the old.
Absolutely love this explanation. As someone else mentioned, this is even better than Tom Scott's explanation on the numberphille channel.
This is such a great explainer video. As somebody who's run into the codepoint slicing problem in Python a lot. This explains so much! Thank You again!
I've watched a few videos today trying to understand these concepts and this is by far the best. Good job
I kinda hoped this video would teach me how to use all the unicode characters in 11min, but it turns out it's just a general overview of how UTF-8 works. This is like going to a math class and learning that addition exists, but not learning how to use the + man.
This is so awesome yet somehow has so few views.
These kind of videos are amazing for people self-studying CS. I hope you have lots more videos like this to make it easier.
Keep up the great work 🤘
Glad you liked it! Any topic suggestions for future videos?
This is the best explaination of Unicode.👏🏼👏🏼👏🏼
Best explanation with a fantastic presentation. Thank you so much
Terrific video dude, definitely deserves far more views
Your tutorial videos are amazing. I decided to go back to creating soft after 16 years. soft soft is so easy to get into, but also offers
no videos were explaining clearly the step of converting the "grapheme" to a "code point" and then to binary they were just assuming that middle step was understood. that's what I was missing apparently thanks great video
Hey thanks for the explanation; I spent some time understanding the Unicode with the tables ASCII table but this explanation is really good. Thanks.
This video packed so much information in such a short amount of time, amazing. Thank you for this content and please keep uploading,
I have waited for this type of video for years, thank goodness you made one!
this is the best video I've seen for how unicode works! Thanks!
wow, best explanation i've seen... short and accurate. Bravo.
8:15 In python3, at least now it can do that with python without any problem.
>>> a="你好"
>>> a[0]
'你'
In python3, every string is in utf8. Using a u"" is a thing in py2
Technically, the internal representation of strings in Python is actually UTF-32 or UCS-4 (they're basically the same), because strings need to be efficiently indexable - as these encoding are uniform-length, that's easy to do. Using utf-8, the lookup of a given character in a string would take longer, because the whole string up to that point would need to be traversed as bytes are not evenly allocated to each character.
To top things off, strings that only contain Latin characters just use ascii anyway internally to reduce waste. That's why when a string is appended to, it creates an entire new string object, as the characters the new string has to encode may be outside the original encoding's range.
Hey, your Sales Incentive payment is all sorted out and good to go!
honestly thought you had a million subs. quality is top notch.
Beautifully Explained
you deserve a fucking award man, I swear, this is the top 1 explanation on TH-cam. NONE of the others are even close to yours.
Thanks for showing actual code.. it made it very easy to understand
Fun fact: most writing systems can express their characters in 2 o 3 UTF-8 bytes (only 2 for the most used ones: extended latin, arabic, cyrilic, greek…). The only outlier is CJK Ideographs (Chinese, Japanese and Korean). There are an insane amount of them so each CJK grapheme carries more meaning (like a word) so it makes sense that they take more bytes in UTF-8 (usually 4 bytes).
ok i finally understand why i need to specify utf 8 when writing to files
Best tutorial on Unicode
Thanks, this video really helped me understand text encoding better.
I'm a programmer so this knowledge is really useful :)
loved this video! So so clear ✨
Such best videos better than 100X bad books, I can't stress too much how good this video is. There are always treasure videos like this on TH-cam, hope OP keeps working, keep letting your knowledge and legacy to influence more people.
wow your Chinese pronuciation is pretty good there @2:24
cleared all my doubts. best explanation on the entire universe :)
thank you so much alex , great and lucid explanation .
WHY? WHY? WHY HAVENT I SEEN YOUR CHANNEL, YOU ARE THE BEST HANDS DOWN, definitely sub + bell (and like)
just the way you explain.. its so good and clear
Yeah, I don`t know what is going on with my channel. Is TH-cam ripping me off? I know the Freedom Fighters [world-wide] are.
Excellent Explanation Man... Understood every bit of it... Many Many Thanks😊❤️
I like Unicode, because its can fit every "not from earth" language character to.
Amazing explanation about Unicode
Such clean video format for explanning clearly a knowledge ! Thank you^^ (by a newcomer in the field of informaticL
May you have a nice day with cookies and tea/milk 🥛🍪
Studying with Alex, the PRO!
Very good explanation. Clearly deserves more views.
i really apreciate your help with dowloanding this software
The greatest unicode video ever
Beautiful man, you're the only one who helped me
This was very informative and clear. Thank you.♥
Brilliant explanation Alex, thank you
Excellent, thank you - helpful and interesting!
Easy to digest explanation! Good job, thanks 👍
Working with strings in python3 has been a godsend. The only time I how think about the encoding now is when opening files.
Absolutely amazing video. Good Job 👍.
TNice tutorials one is really good, among all other basics videos
Some video games use a custom encoding/decoding systems. Like Resident Evil 4. I learned that while trying to translate the game to Arabic.
Interesting stuff.
at 1:49 the arabic is rendered incorrectly. it shows each letter in the isolated form instead of the connected forms.
and it's written from left to right instead of right to left
happens every time lol, i remember some big project did the same thing recently
It would have been nice if you explained how each collection of graphe,es are organized and how the UTF-8 parser traverses the tree of each set.
Hi! This is Gold! Please do more videos like this. especially the terminal colors
This is so nicely explained
So much informative video, thank you for creating such a video, and also salute to your effort for making this video. 👍 ❤
Excellent video! Explained everything in a way that I could understand very well, thank you :)
Thanks man!! You've earned my respect
Recently I was rewriting my python program which prints all the Unicode Characters (without the combining ones) within a range of Hexadecimal characters or chosen unicode blocks. You can see it on my channel but know that i am a beginner and i am aware of how i can improve it because I created it a long time ago.
Best video about unicode
Hi, I have some awesome news that will bring a smile to your face!
Very good explanation , thank you for your work!
Good Good i have really suffered in this TH-cam before i met you
Great video , made me understand finally.
Very good video! Thank you a lot!
Thanks a lot! Best explanation ever
Thank you.. this was easy to understand with your explanation
great content, it's much more clear now. Thxxxxxxx
Quality content, awesome. Post more these kind of great explainable video
subbed because of this wonderful explanation
thanks
yes it works brother ! many thanks
Thank you so much for this!
the len string function has been updated to be universally aware of UTF encoding