1:34 Fun fact: separating bytes from strings was the most important major breaking change between Python 2 and Python 3. Trying to keep strings as byte-encoded led to all kinds of unfortunate trouble in Python 2, which could not be fixed without sacrificing backward compatibility. And they thought, while they were breaking things anyway, they might as well fix a few other things in a cleaner, non-backward-compatible way while they were at it.
When I started learning Rust, this was something that actually comes up quite a bit, since you can't iterate over a string object (you don't necessarily know its encoding at compile time). It was the first time I realized that the difference between ascii, utf, and others is actually really important!
This seems mistaken. Any decodable encoding is iterable, and Rust proponents keep bragging about compile time checks and zero overhead abstractions. A quick look up of Rust's std::string::String does define it as encoded using UTF-8, and it inherits the iteration method chars() from str. So it appears you're talking about something else, perhaps an equivalent to Python's bytes type.
@@0LoneTech Remember not everything is an object in Rust. Rust has two "strings" to be aware of. The first is a stack "str" which is a primitive type stored on (you guessed it) the stack as bytes. This is the default behavior when declaring a string in Rust. It might be "iterable" at this point, but I don't think the memory safety checks can hold true if you start manipulating values on the stack. Rather, Rust makes you create a slice of "str" on the heap as a "String" struct before doing manipulation.
I think the comment is just weirdly stated. String itself does not implement the Iterator trait (aka there is no "default" way to iterate a String), instead it makes you choose between characters or bytes by using the chars() or bytes() method that return the appropriate iterator.
@@mr.bulldops7692 You're talking about mutability, a separate subject in both Rust and Python. This is distinct in turn from owned or borrowed, the prime difference between Rust's String and str (with lifetimes guarding stack frame ownership, iirc). &mut str does let you modify the data, irrespective of where it is stored, but one should take care not to violate UTF-8 encoding when doing so. The borrow checker should prevent breaking character boundaries while slices depend on them. I expect string literals to be in rodata, as 'static str, neither mutable nor in stack or heap.
The way you phrased it makes it seem like Rust doesn't know the encoding of a String/str at compile-time, which is bonkers. The type itself enforces the variant that it holds UTF-8 data. It's not encoding that's unknown at runtime, it's the grapheme clusters. Indexing is supposed to be O(1) for example, but you don't know what str[5] character will be in O(1), since maybe an emoji is in b/w which will take more than one byte, and so there's no direct mapping b/w index and character position. If you just want to assume byte array, just call .to_bytes() on it.
Windows encodings are a real nightmare. There are the OEM/MS-DOS codepages used by the console which make almost impossible to consistently write non-English characters from a .bat script. Then there are the "ANSI" codepages which are used by the Win32 functions accepting strings as char pointers (e.g. MessageBoxA). It is usually Windows-1252 in western countries which is a slightly incompatible variant of ISO 8859-1 (also known as "Latin1"). Then there are the "Unicode" strings/MBCS/wchar_t pointers which are actually UTF-16 (even MS documentation states wrongly that "Unicode is a 16-bit character encoding"), meaning that Emojis will probably work in some places and not in others (try calling MessageBoxW with an emoji...). Except not really because in some cases it is UCS-2 instead of UTF-16 (another slightly incompatible variant). BTW, at least until recently you needed to add the BOM character to make stuff like notepad to recognize a UTF-16 file. Note that NONE of those encodings are UTF-8.
Perhaps some of the confusion comes from this: Bytes are interpreted as bytes, but you type them as if they were string literals. So then, how does the string get transformed into bytes? Doesn't it have some sort of implicit encoding when you type it in your IDE?
It's crazy how much you come across decoding/encoding issues in the wild. I sometimes work with large text datasets with mixed encodings, sometimes even in the same line! The worst is that if you try and decode with the wrong encoding it can raise a runtime error, so I ended up writing a short program with a bunch of try/excepts for the different possibilities (utf-8 first of course). I did the same thing when I worked in C and Tcl. Gotta be a better way...
Chardet is a helpful library for trying to guess the most likely encoding. However, if you've got a single string with mixed encodings, then that might not be helpful.
5:24 "UTF-8 is by far the most common encoding across all programming languages." I don't think this is actually true. UTF-8 is the most common encoding for on disk and on the web text, but programming language implementations will often internally work in a different encoding than the wire/disk encoding. Both in Java and DotNet the String type is internally implemented using UTF-16, not UTF-8.
@@lawrencedoliveiro9104 Voluntary or not, utf-16 is now deeply imbued into every NT-derived Windows computer out there. The ubiquitous windows-1252 et al is only what we see on "the surface" within some GUI apps and the command window. NT used to have "16-bit Unicode", but after Unicode expanded past the BMP they redefined it to be utf-16 instead. I wonder how many bugs are still hiding in there that don't actually handle the utf-16 "escapes" correctly and just assume every character is 16 bits...
Wow really informative. I wrote most of a project in windows, started using it in a Linux Google cloud VM, but I realized some of my data in a csv file was invalid. In the interest of getting a proof of concept out quick, I just quickly wrote a script in the VM that opens the file as a pandas dataframe, removes the invalid rows, and stores it as a csv file again. Except when I went to open this new file before giving it to my ML algorithm, it kept telling me the file was corrupted. I couldn't understand it, I was at a total loss, and I ended up just writing another hacky solution in which if I encountered an error loading one of the rows during the training process, I would just default to loading the first row instead. This makes total sense that this could have been the problem. Thanks James!
I’m sure Pandas has ways to hook into the loading/saving process. Python has options in decoding as to how to treat invalid byte sequences: for example, you could ignore them, or replace them with some marker character.
@@lawrencedoliveiro9104 Yep, in this case pandas.read_csv has arguments encoding, encoding_errors and on_bad_lines. One guess at a cause of the corruption might be how Windows NT Notepad silently injects an invalid BOM into UTF-8 files.
5:25 Not just the most popular, but some languages, including Python, have embraced Unicode to the point that identifiers can contain any Unicode characters that are classed as “letters”. So for example while “in” is a reserved word, “ın” is not, and can be used as an identifier.
Does it warn you about doing this? This rings a bell... I think some language out there actually does warn you if you do silly stuff like this. Maybe it was Rust or c++? Not sure.
It can be fun, then, to mix otherwise identical characters from the Latin, Greek and Cyrillic alphabets. You can write dеf instead of def, for example.
yea newer languages such as GO ([]byte) and ZIG ([]const u8) use a slice of bytes to interchangeably represent strings which makes sense from a C/C++ perspective considering that strings in C/C++ are essentially just an array of characters and characters are essentially uint8_t (bytes)
Yeah this is an important choice that lower level languages make. "Strings" in those languages are more like the bytes object in Python, a contiguous container of bytes with stringlike functions. If you want true unicode support you have to use some external lib, which makes sense in performance driven languages because parsing utf8 at runtime is a huge performance penalty.
so this solved a thing that I didn't think about so you know the *base64.b64decode* function (import base64) so when you decode a base 64 string the output is b'(decoded content)' which is as I just found out a byte formatted string before what I was doing was this (mind the naming schema) base64_decoded= b'some|text|here' str_base64_decoded= str(base64_decoded) and then str_base64_decoded[2:-1] (which is the same as slice() which is formatted as (start, stop, step) so what that did was remove the *b'* and then also removes the ending *'* to give *some|text|here* so yeah knowing byte formatted strings exist helps as instead I can just do this base64_decoded.decode() which will get me the same output *some|text|here* thanks for reading my weird experience, have a good day
@@AssemblyWizard so doing a test I see what you mean the first row is with the byte to string (with str()) and the bottom one is .decode() "\xe2\x82\xac" byte to string then cut "€" using built in func which the main difference is .decode() being able to properly represent characters that byte strings can't thanks for letting me know the code I used is down bellow import base64 base_decode = base64.b64decode("4oKs") str_decoded_cut = str(base_decode)[2:-1] base_decode_builtin_func = base_decode.decode() print(f'"{str_decoded_cut}" byte to string then cut
"{base_decode_builtin_func}" using built in func')
Great video! But is not it weird that python's string encode() and open() use different default encodings? I've asked chat GPT and it says Python 3.10 has 17 built in functions and class methods that use encoding parameter and there's 5 default values utf-8, None (system's default), latin-1, ascii, utf-16. This is bad, right?
ChatGPT is the best storyteller humankind has ever seen. To distinguish the fake from the truth, possible it's not. Do not put your faith on it if your question might be less popular, it generalizes everything.
Yes it is very weird, and there were historical reasons for doing it that are somewhere between no longer very relevant and a mistake. That's why PEP 686 is finally switching the default to utf8, but since this is big change they have to wait until 3.15!
5:50 if utf-8 character has 4 bytes shouldnt have there been padding bytes after/before every byte from 65 to 68 ? 0,0,0,65, 0,0,0,66, 0,0,0,67, 0,0,0,68, 201,184,240,159 ...
Great question and indeed the solution is that utf-8 is a variable-length encoding. The way this works is by encoding the number of total bytes in the character within the first byte. If the first byte starts like: 0xxxxxxx -> 1 total byte 110xxxxx -> 2 total bytes 1110xxxx -> 3 total bytes 11110xxx -> 4 total bytes. In particular, since ascii values are 0-127, they all start out 0xxxxxxx and hence all ascii values are encoded in a single byte in uft-8. Clever! Read more here: en.wikipedia.org/wiki/UTF-8
I absolutely love your videos. Regardless of my familiarity with a topic, every video seems to have some piece of information that I would not have discovered on my own. I never knew that files were encoded with the system encoding unless specified. It has never been an issue, but I know that one day it will be and without this knowledge, I would have really struggled to identify the issue. Future me really appreciates your hard work.
Great to hear! And I was also surprised when I ran the example and found out that my default encoding was not utf-8. Then I remembered I record my videos on Windows!
4:19 Just a note that, in Python, the len() function is counting “code points”, not characters. So strings are really being interpreted as sequences of code points, not of actual Unicode characters (which can be encoded in multiple code points).
This isn't particularly a video about the specifics of Unicode, but since you mention it, a "character" as defined by the Unicode standard is the same as a code point, and this is the same way that I use the term in this video. You may be confusing the term "character" with a "glyph", which is a shape that is rendered as a representation of one or more characters. Various relationships may exist between character and glyph: a single glyph may correspond to a single character or to a number of characters, or multiple glyphs may result from a single character. Python's len function counts both characters and code points because these are the same, but it does not count glyphs. I refer you to Section 2.2 "Characters, Not Glyphs" in the Unicode standard for further explanation.
@@mCoding No. Consider a character with diacritic marks, like for example “ä”. This has its own code point U+00E4, but it can also be represented as U+0061 (“a”) followed by the combining diacritic mark U+0308. The most common character-plus-diacritic combinations have their own assigned code points, but not every combination can be represented this way. Hence the need for multiple code points to represent a character.
As I was searching the interwebs as to what a type 'byte' was and how to convert it to a string, my YT refreshed and there was this video at the top of my subscription, 4 minutes old. This timing was apropos.
With all the stuff Python gladly broke in the 2to3 switch I'm shocked bytestrings stayed around, all they do is confuse people for minimal convenience. Shoulda swapped it out with a fixed-length bytearray implementation, that way nobody gets confused about the intended purpose.
How does Python store some string, e.g. test: str = "some-test-string" in memory? I mean, if it is of "bytes" type and encoded in UTF-8 it is clear what bytes will be written in memory, but I simply cannot logically explain to me how does Python handle non-encoded strings, they should live somewhere in memory, while the programming is executing, but how will they be stored? For instance the example that you store emoji in a variable, how is it stored in memory, before you encode it and print it?
One thing that's missing I feel is that a bytes is actually a list of ints between 0 and 256 nothing more. It makes sense but it was like an ah moment for me
What i know is that utf-8 is also 8-bit length so how he knows that he should interpret the 4 bytes as a emoji instead of some other 4 8-bit characters? Shouldn’t we use utf-32?
I have been using tree sitter for a language model dataset. I use the start_byte and end_byte to cut out a function and replce it with generation for the benchmark. I spend a few hours hunting down some offset issues... And it was due to difference in len for str vs bytes, also indexing. So i do a lot of encode, slice, decode. and its awful. I woild love to simply use the byte index to slice a str.
For the actual size in memory it's actually up to the implementation, they can differ due to things like small string optimization, ascii-only optimization, cached properties, and a few other things. When you write a string to disk, it is always converted to bytes first (it is done automatically in text mode) so in that sense the storage size is the same. However, the "length" of a string is the number of characters, which can differ from the number of bytes because some characters can take multiple bytes (like the smiley).
Let me amend that. The valid ranges for Unicode code points are [0 .. 0xD7FF] and [0xE000 .. 0x10FFFF]. The values in the gap are called the “High Surrogates” and “Low Surrogates”, and are reserved for representing UTF-16 encodings. Which nobody should be using any more.
The bytes literal syntax b"...." always uses ascii encoding and does not allow non-ascii characters like "😉" in the literal. If you want to include bytes (0-255 allowed) outside the ascii range (0-127 allowed), then you have to feed it an iterable of integers like bytes([255, 255, 255]) instead of using the literal syntax.
This makes me glad I only ever work on Linux systems. Utf-8 everywhere. I would have never even considered python using anything other than utf-8 when opening a file in text mode. Although, I also didn't know encode and decode could be used without an argument. I always specified utf-8 and will continue to do so.
Be warned! The standard library json.write() function has an "ensure_ascii" variable that for some reason defaults to True. If you want to save data that's not just standard latin text you have to set it to False. I guess we have to wait for 3.15 for that to change...
@@0LoneTech And that makes it unreadable in a text editor. I guess it's cool that all the data is still there, but is there a reason to not keep the data _and_ have it readable? What doesn't support UTF-8 these days?
@@volbla 6:08 is one example, likely Windows-1252. The default ensure_ascii format will function through that, utf-8 won't. JSON does not support indicating encoding like e.g. HTML or HTTP. So if it was stored in a file, the default system encoding is the only suggestion.
Heads up: Unicode: It's like a dictionary of characters. Each character has a unique entry and a value which identifies it, aka code-point. A Code-point is a 4 bytes value. An encoding (there are several), encodes this unique code-point(4 bytes)to a sequence of bytes(variable sized), so as to save space. Eg. A utf8 encoded character can use 1/2/3/4 bytes depending on its code-point. Similarly, a utf-16 encoded character can use 2/4 bytes. Why utf8 is so popular you might ask? Reason is backward compatiblity to ASCII, all the ascii characters encoded to utf8 shares the same "value" when encoded to ASCII, E.g. 'A' is 65 in both encodings. All ascii chars uses only 1 byte in utf8. Why utf16, well utf8 cannot represent all the unicode chars, there are some chars that have code-points outside the range of utf8.
@@benhetland576 Just checked and it's looks like utf8 does support all code points according to wikipedia, but I'm not sure if it's correct. I saw this warning when I was working with powershell script, it needed input encoded specifically in utf-16LE since it mentioned that this supports 'all' code points. Again, not sure why it might say that.
@@tusharsnn The encoding that utf-8 uses theoretically allows a max of 7 octets (or bytes if you like), in which case the first octet would start with 7 ones followed by a 0, i.e, 0xFE. The next 6 octets each encode 6 bits for a total of 36 bits, and only 21 bits are needed to cover all possible Unicode codepoints (17 "planes" of 65536 codepoints each). 4 octets encode (8 - 5) + 3 × 6 = 21 bits, so that is the longest octet sequence ever needed to encode a single Unicode codepoint. There are byte sequences that are not valid utf-8 for several reasons (even single octets like a 0xFF), but not vice versa.
There are also characters with multiple code points, such as latin V and roman numeral V, and characters with distinct glyphs such as Han unified ones, and combined characters that may be separable like ä, and the classification and ordering of characters is language dependent. Text is messy. Unicode does not provide the tools needed to compare a Chinese and Japanese phrase in one text, by design. History and politics (not necessarily national) are involved.
Thanks, James.Very concise, well-informed and well-executed . I especially like the grounded references to Python 3.1x.x My inbox needs a vaccine for "Python 4 .*" titles. I might take a look at the meta on this vid to see if I can spot such effusion for my personal ytube feed.
When you ask python to decode using utf8 (whether you specify the encoding specifically or whether python just does that by default), you are asking python to interpret the bytes according to the unicode standard, and the unicode standard specifies that that exact sequence of bytes means "smiling face" or some other similar description. It is then up to the author of the font you are using to create a glyph (the picture of an actual smiley) to draw whenever you try to display the smiley as a character. This allows for different sets of smileys, like the Apple smileys vs the Microsoft smileys, choosing between them is simar to choosing a different font.
Yesterday I wrote a self-extracting installer script. Today I see this. I found it easiest to write the installer file as a string and then write the files it contains as bytes that are encoded as utf-8, especially if they are binary files. Writing the whole installer to file as bytes caused me trouble with the python interpreter.
Excellent video, thank you. I've had a couple of issues where I've had to use the IO and locale libraries to "fix" encoding shenanigans, but I think if I revisited those lines I'd now have an actual understanding of what was happening, how the changes worked and, most importantly, how /to do it better/.
Great! So thankful, it cleared my confusion (I still didn’t go to Stack overflow for it 😅😂) ... May I just have curious question for an end? What might be the reason for anyone to have system not having UTF-8 as a default? (why not?)
Legacy reasons. Before Unicode--indeed, for a long while after--there were these things called “national character sets”. In fact, there is likely still a large collection of text stored in these legacy encodings.
What a coincidence! I just tried to survey the difference between them and found this video that just came out! 😀 So... why isn't your computer in UTF-8? 🤣
UTF8 ≠ Unicode I was expecting you to explain the difference, since it's super related to bytes vs. str, but instead you said these terms commonly refer to the same thing 😓
FWIW here's the difference: Unicode usually refers to the conversion between characters and numbers (ord/chr in python), and UTF8 is the conversion between these numbers and bytes. (although Unicode is technically the name for the entire standard, including both UTF8 and the mapping between characters and numbers)
I completely agree that UTF-8 != Unicode. However, it is widespread and commonplace for developers (and others) to colloquially use the word "Unicode" interchangeably with UTF-8 as most people don't bother with technical distinctions and it is usually clear from context which conversion is meant.
Bytes are mostly just useful if you mean to interpret them as something very specific that's not text. For example encryption keys or raw image data. If you're writing or handling text you should go with strings.
By default, we should be good to know why we use string in Python. But, is there a practical use case for us to use bytes ? That may helps us to differentiate the uses between string and bytes here.
It's simply one level less of abstraction. Bytes hold arbitrary data and can be used in I/O, like storing or transmitting. String is for when data is text, while other formats could be handled with e.g. struct or ctypes. This video had a couple of examples decoding some bytes as little or big endian integers. It wouldn't make sense to pass a string to zlib.decompress() for instance.
Can you make a video on sets. How they are implemented? What is the time/space complexity of adding elements, creation from list, removing elements and others? Why it is at all in the language? Is using lists faster in specific situatutions?
Small hint: Instead of writing `encoding="utf-8"`, you can just write `encoding="u8"`, which saves you up to 3 keystrokes! Check the Python doc and you can see that `u8` is just another name for `utf-8`.
even i knew it already since i literaly had to google it a hundret times because i found it funny to even ask the question ^^ like bites are just a string of 1 and 0 just that every 1 and 0 of your string takes at least 8 1´s and 0´s and thus is at least 8 times your memory used utf16 utf32. Let me tell you if came to the idea just useing str ('10010011' ) work with actual biits since you more comfortable with them then bytearrays dont its so painfuly slow i know cause i tryed it -.-
1:34 Fun fact: separating bytes from strings was the most important major breaking change between Python 2 and Python 3. Trying to keep strings as byte-encoded led to all kinds of unfortunate trouble in Python 2, which could not be fixed without sacrificing backward compatibility.
And they thought, while they were breaking things anyway, they might as well fix a few other things in a cleaner, non-backward-compatible way while they were at it.
That’s actually super interesting 😮 thanks for the info
When I started learning Rust, this was something that actually comes up quite a bit, since you can't iterate over a string object (you don't necessarily know its encoding at compile time). It was the first time I realized that the difference between ascii, utf, and others is actually really important!
This seems mistaken. Any decodable encoding is iterable, and Rust proponents keep bragging about compile time checks and zero overhead abstractions. A quick look up of Rust's std::string::String does define it as encoded using UTF-8, and it inherits the iteration method chars() from str. So it appears you're talking about something else, perhaps an equivalent to Python's bytes type.
@@0LoneTech Remember not everything is an object in Rust. Rust has two "strings" to be aware of. The first is a stack "str" which is a primitive type stored on (you guessed it) the stack as bytes. This is the default behavior when declaring a string in Rust. It might be "iterable" at this point, but I don't think the memory safety checks can hold true if you start manipulating values on the stack. Rather, Rust makes you create a slice of "str" on the heap as a "String" struct before doing manipulation.
I think the comment is just weirdly stated. String itself does not implement the Iterator trait (aka there is no "default" way to iterate a String), instead it makes you choose between characters or bytes by using the chars() or bytes() method that return the appropriate iterator.
@@mr.bulldops7692 You're talking about mutability, a separate subject in both Rust and Python. This is distinct in turn from owned or borrowed, the prime difference between Rust's String and str (with lifetimes guarding stack frame ownership, iirc). &mut str does let you modify the data, irrespective of where it is stored, but one should take care not to violate UTF-8 encoding when doing so. The borrow checker should prevent breaking character boundaries while slices depend on them.
I expect string literals to be in rodata, as 'static str, neither mutable nor in stack or heap.
The way you phrased it makes it seem like Rust doesn't know the encoding of a String/str at compile-time, which is bonkers. The type itself enforces the variant that it holds UTF-8 data. It's not encoding that's unknown at runtime, it's the grapheme clusters. Indexing is supposed to be O(1) for example, but you don't know what str[5] character will be in O(1), since maybe an emoji is in b/w which will take more than one byte, and so there's no direct mapping b/w index and character position. If you just want to assume byte array, just call .to_bytes() on it.
Windows encodings are a real nightmare.
There are the OEM/MS-DOS codepages used by the console which make almost impossible to consistently write non-English characters from a .bat script.
Then there are the "ANSI" codepages which are used by the Win32 functions accepting strings as char pointers (e.g. MessageBoxA). It is usually Windows-1252 in western countries which is a slightly incompatible variant of ISO 8859-1 (also known as "Latin1").
Then there are the "Unicode" strings/MBCS/wchar_t pointers which are actually UTF-16 (even MS documentation states wrongly that "Unicode is a 16-bit character encoding"), meaning that Emojis will probably work in some places and not in others (try calling MessageBoxW with an emoji...). Except not really because in some cases it is UCS-2 instead of UTF-16 (another slightly incompatible variant). BTW, at least until recently you needed to add the BOM character to make stuff like notepad to recognize a UTF-16 file.
Note that NONE of those encodings are UTF-8.
Remember that MS-DOS was originally created for the IBM PC, so it had to incorporate the whole IBM “code page” concept.
Perhaps some of the confusion comes from this: Bytes are interpreted as bytes, but you type them as if they were string literals. So then, how does the string get transformed into bytes? Doesn't it have some sort of implicit encoding when you type it in your IDE?
yes, ascii. a=1, b=2, etc
@@jacobgoldsmith7651 that’s… not the ascii table?
Everything is Unicode these days. When you convert between Unicode strings and bytes, the default decoding/encoding is “utf-8”.
@@jacobgoldsmith7651 In ASCII, a = 97, b = 98, etc, and A=65, B=66, etc.
I don't know for sure, but I'd imagine that it either uses the literal bytes that whatever text editor you used decided to encode the bytes literal as
Looking forward to "from __future__ import default_encoding".
A very likely possibility. Maarten called it first! My guess is `from __future__ import utf8`
It's crazy how much you come across decoding/encoding issues in the wild. I sometimes work with large text datasets with mixed encodings, sometimes even in the same line! The worst is that if you try and decode with the wrong encoding it can raise a runtime error, so I ended up writing a short program with a bunch of try/excepts for the different possibilities (utf-8 first of course). I did the same thing when I worked in C and Tcl. Gotta be a better way...
*Mixed* encodings! That's nightmare fuel for developers if I ever heard it!
Chardet is a helpful library for trying to guess the most likely encoding. However, if you've got a single string with mixed encodings, then that might not be helpful.
Nightmare indeed. One rare program I've seen handling it decently is mlterm with mixes like ISO 8859-1 and EUC-JP.
I have dealt with that just recently. Absolutely terrible.
@@Bobbias thank you I'll check it out!
5:24 "UTF-8 is by far the most common encoding across all programming languages." I don't think this is actually true. UTF-8 is the most common encoding for on disk and on the web text, but programming language implementations will often internally work in a different encoding than the wire/disk encoding. Both in Java and DotNet the String type is internally implemented using UTF-16, not UTF-8.
UTF-16 is an unfortunate hangover from the early days of Unicode. Nobody uses it voluntarily any more.
@@lawrencedoliveiro9104 Voluntary or not, utf-16 is now deeply imbued into every NT-derived Windows computer out there. The ubiquitous windows-1252 et al is only what we see on "the surface" within some GUI apps and the command window. NT used to have "16-bit Unicode", but after Unicode expanded past the BMP they redefined it to be utf-16 instead. I wonder how many bugs are still hiding in there that don't actually handle the utf-16 "escapes" correctly and just assume every character is 16 bits...
@@benhetland576 And that’s why you wouldn’t choose to use it.
Instructions unclear: Slapped the like button an even number of times.
Wow really informative. I wrote most of a project in windows, started using it in a Linux Google cloud VM, but I realized some of my data in a csv file was invalid.
In the interest of getting a proof of concept out quick, I just quickly wrote a script in the VM that opens the file as a pandas dataframe, removes the invalid rows, and stores it as a csv file again. Except when I went to open this new file before giving it to my ML algorithm, it kept telling me the file was corrupted. I couldn't understand it, I was at a total loss, and I ended up just writing another hacky solution in which if I encountered an error loading one of the rows during the training process, I would just default to loading the first row instead.
This makes total sense that this could have been the problem. Thanks James!
A true war story from the field! This is exactly what can happen with mixed encodings and I'm glad this helped figure out the problem!
I’m sure Pandas has ways to hook into the loading/saving process. Python has options in decoding as to how to treat invalid byte sequences: for example, you could ignore them, or replace them with some marker character.
@@lawrencedoliveiro9104 Yep, in this case pandas.read_csv has arguments encoding, encoding_errors and on_bad_lines.
One guess at a cause of the corruption might be how Windows NT Notepad silently injects an invalid BOM into UTF-8 files.
5:25 Not just the most popular, but some languages, including Python, have embraced Unicode to the point that identifiers can contain any Unicode characters that are classed as “letters”. So for example while “in” is a reserved word, “ın” is not, and can be used as an identifier.
Does it warn you about doing this? This rings a bell... I think some language out there actually does warn you if you do silly stuff like this. Maybe it was Rust or c++? Not sure.
@@BenjaminWheeler0510 It’s a feature, not a bug.
@@BenjaminWheeler0510 it only warns you if it's not a "letter" character (emoji,etc), katakana/hiragana/etc works as variable name without warning
It can be fun, then, to mix otherwise identical characters from the Latin, Greek and Cyrillic alphabets. You can write dеf instead of def, for example.
@@BenjaminWheeler0510 Some IDEs warn you about it, for example vscode has a warning about non standart characters
yea newer languages such as GO ([]byte) and ZIG ([]const u8) use a slice of bytes to interchangeably represent strings
which makes sense from a C/C++ perspective considering that strings in C/C++ are essentially just an array of characters
and characters are essentially uint8_t (bytes)
Yeah this is an important choice that lower level languages make. "Strings" in those languages are more like the bytes object in Python, a contiguous container of bytes with stringlike functions. If you want true unicode support you have to use some external lib, which makes sense in performance driven languages because parsing utf8 at runtime is a huge performance penalty.
so this solved a thing that I didn't think about
so you know the *base64.b64decode* function (import base64)
so when you decode a base 64 string the output is b'(decoded content)'
which is as I just found out a byte formatted string
before what I was doing was this (mind the naming schema)
base64_decoded= b'some|text|here'
str_base64_decoded= str(base64_decoded)
and then str_base64_decoded[2:-1] (which is the same as slice() which is formatted as (start, stop, step)
so what that did was remove the *b'* and then also removes the ending *'*
to give *some|text|here*
so yeah knowing byte formatted strings exist helps as instead I can just do this
base64_decoded.decode()
which will get me the same output
*some|text|here*
thanks for reading my weird experience, have a good day
Won't always give the same output, try decoding the base64 "4oKs" (that's a lowercase O not a zero), and then compare str with slicing vs decode
@@AssemblyWizard so doing a test I see what you mean
the first row is with the byte to string (with str()) and the bottom one is .decode()
"\xe2\x82\xac" byte to string then cut
"€" using built in func
which the main difference is .decode() being able to properly represent characters that byte strings can't
thanks for letting me know
the code I used is down bellow
import base64
base_decode = base64.b64decode("4oKs")
str_decoded_cut = str(base_decode)[2:-1]
base_decode_builtin_func = base_decode.decode()
print(f'"{str_decoded_cut}" byte to string then cut
"{base_decode_builtin_func}" using built in func')
Huh, didn't know about that system encoding. Very much agree with PEP 686.
Decoded the mystery in a few minutes, thank you! ☺
When you're completely unsure what the encoding of any file that you're processing is, the chardet package is really helpful.
Great tip!
Discord gang
Best gang!
> Stack Overflow?
@@swizice yuuuup'
mcoding != encoding. Amazing. 😂
Great video! But is not it weird that python's string encode() and open() use different default encodings? I've asked chat GPT and it says Python 3.10 has 17 built in functions and class methods that use encoding parameter and there's 5 default values utf-8, None (system's default), latin-1, ascii, utf-16. This is bad, right?
Sounds like an absolute mess
Why ask ChatGPT instead of reading the documentation?
ChatGPT is the best storyteller humankind has ever seen. To distinguish the fake from the truth, possible it's not. Do not put your faith on it if your question might be less popular, it generalizes everything.
Yes it is very weird, and there were historical reasons for doing it that are somewhere between no longer very relevant and a mistake. That's why PEP 686 is finally switching the default to utf8, but since this is big change they have to wait until 3.15!
Checking help(open), it says the default encoding is taken from your locale. But I always set my locale to something UTF-8-based anyway, so no biggie.
5:50 if utf-8 character has 4 bytes shouldnt have there been padding bytes after/before every byte from 65 to 68 ? 0,0,0,65, 0,0,0,66, 0,0,0,67, 0,0,0,68, 201,184,240,159 ...
No. UTF-8 is variable-length. In particular, all the values in the range 0 .. 127 fit in a single byte.
@@lawrencedoliveiro9104 variable length explains. i dont get how it can be variable length and work but i will google it. thx
Great question and indeed the solution is that utf-8 is a variable-length encoding. The way this works is by encoding the number of total bytes in the character within the first byte. If the first byte starts like:
0xxxxxxx -> 1 total byte
110xxxxx -> 2 total bytes
1110xxxx -> 3 total bytes
11110xxx -> 4 total bytes.
In particular, since ascii values are 0-127, they all start out 0xxxxxxx and hence all ascii values are encoded in a single byte in uft-8. Clever! Read more here: en.wikipedia.org/wiki/UTF-8
@@bartlomiejodachowski Hopefully you found your answer, but if not, there was a very good Computerphile video featuring Tom Scott about UTF8
@quillaja i didint delete my comment in case someone had simmilar question. i have alredy studied encodings, but thx for your response.
I absolutely love your videos. Regardless of my familiarity with a topic, every video seems to have some piece of information that I would not have discovered on my own. I never knew that files were encoded with the system encoding unless specified. It has never been an issue, but I know that one day it will be and without this knowledge, I would have really struggled to identify the issue. Future me really appreciates your hard work.
Great to hear! And I was also surprised when I ran the example and found out that my default encoding was not utf-8. Then I remembered I record my videos on Windows!
@@mCoding yep, the "we do things different because.... (Usually) bad reasons." OS. As a longtime windows user, it's so damn frustrating sometimes.
Thank God for UTF-8
4:19 Just a note that, in Python, the len() function is counting “code points”, not characters. So strings are really being interpreted as sequences of code points, not of actual Unicode characters (which can be encoded in multiple code points).
This isn't particularly a video about the specifics of Unicode, but since you mention it, a "character" as defined by the Unicode standard is the same as a code point, and this is the same way that I use the term in this video. You may be confusing the term "character" with a "glyph", which is a shape that is rendered as a representation of one or more characters. Various relationships may exist between character and glyph: a single glyph may correspond to a single character or to a number of characters, or multiple glyphs may result from a single character. Python's len function counts both characters and code points because these are the same, but it does not count glyphs. I refer you to Section 2.2 "Characters, Not Glyphs" in the Unicode standard for further explanation.
@@mCoding No. Consider a character with diacritic marks, like for example “ä”. This has its own code point U+00E4, but it can also be represented as U+0061 (“a”) followed by the combining diacritic mark U+0308.
The most common character-plus-diacritic combinations have their own assigned code points, but not every combination can be represented this way. Hence the need for multiple code points to represent a character.
As I was searching the interwebs as to what a type 'byte' was and how to convert it to a string, my YT refreshed and there was this video at the top of my subscription, 4 minutes old. This timing was apropos.
Yet another informative and well put video. Thanks!
With all the stuff Python gladly broke in the 2to3 switch I'm shocked bytestrings stayed around, all they do is confuse people for minimal convenience. Shoulda swapped it out with a fixed-length bytearray implementation, that way nobody gets confused about the intended purpose.
Hey James, I’d love to see a video showcasing how to use the Textual package. It’s really neat, and fits your style.
How does Python store some string, e.g. test: str = "some-test-string" in memory? I mean, if it is of "bytes" type and encoded in UTF-8 it is clear what bytes will be written in memory, but I simply cannot logically explain to me how does Python handle non-encoded strings, they should live somewhere in memory, while the programming is executing, but how will they be stored? For instance the example that you store emoji in a variable, how is it stored in memory, before you encode it and print it?
One thing that's missing I feel is that a bytes is actually a list of ints between 0 and 256 nothing more. It makes sense but it was like an ah moment for me
Instructions not specific, slapped like button 256 times.
Its crazy how one day I am wondering about something and a week later you have a great video on it. Thanks for another great one!
Great to have you watching!
What i know is that utf-8 is also 8-bit length so how he knows that he should interpret the 4 bytes as a emoji instead of some other 4 8-bit characters? Shouldn’t we use utf-32?
I have been using tree sitter for a language model dataset. I use the start_byte and end_byte to cut out a function and replce it with generation for the benchmark.
I spend a few hours hunting down some offset issues... And it was due to difference in len for str vs bytes, also indexing. So i do a lot of encode, slice, decode. and its awful. I woild love to simply use the byte index to slice a str.
The encoding type changing depending on the system's configuration is nightmare fuel.
6:12 ohw, that was the thing the pylance/mypy was shouting at me "encoding not specified"
but how to convert for binary shellcode in python3? i have a chroot python2 version for that
Am I right in assuming the storage size the same between string and byte?
For the actual size in memory it's actually up to the implementation, they can differ due to things like small string optimization, ascii-only optimization, cached properties, and a few other things. When you write a string to disk, it is always converted to bytes first (it is done automatically in text mode) so in that sense the storage size is the same. However, the "length" of a string is the number of characters, which can differ from the number of bytes because some characters can take multiple bytes (like the smiley).
Unicode “code points” can officially have any value in [0 .. 0x10FFFF]. That means a single code point could fit in 21 bits.
Let me amend that. The valid ranges for Unicode code points are [0 .. 0xD7FF] and [0xE000 .. 0x10FFFF]. The values in the gap are called the “High Surrogates” and “Low Surrogates”, and are reserved for representing UTF-16 encodings. Which nobody should be using any more.
wow you learn something new everydaay! had no idea that python has a built in way to do intersections 1:00
very good explanation, Cleared my head :)
2:17 Most humble ad read ever lol
A message from our sponsor: (quiet voice) me.
interesting theme. waiting for your next video :)
One question, i still have: x = b"Hello World 😉" will then automaticly convert my string into bytes using the system encoding as default?
The bytes literal syntax b"...." always uses ascii encoding and does not allow non-ascii characters like "😉" in the literal. If you want to include bytes (0-255 allowed) outside the ascii range (0-127 allowed), then you have to feed it an iterable of integers like bytes([255, 255, 255]) instead of using the literal syntax.
This makes me glad I only ever work on Linux systems. Utf-8 everywhere.
I would have never even considered python using anything other than utf-8 when opening a file in text mode.
Although, I also didn't know encode and decode could be used without an argument. I always specified utf-8 and will continue to do so.
Incredible explanation
I needed this explanation 3y ago :/
One of the best explanations ever.
Be warned! The standard library json.write() function has an "ensure_ascii" variable that for some reason defaults to True. If you want to save data that's not just standard latin text you have to set it to False.
I guess we have to wait for 3.15 for that to change...
ensure_ascii generates \u escape sequences. It makes the JSON ASCII compatible, it does not alter the data contained within strings.
@@0LoneTech And that makes it unreadable in a text editor. I guess it's cool that all the data is still there, but is there a reason to not keep the data _and_ have it readable? What doesn't support UTF-8 these days?
@@volbla 6:08 is one example, likely Windows-1252. The default ensure_ascii format will function through that, utf-8 won't.
JSON does not support indicating encoding like e.g. HTML or HTTP. So if it was stored in a file, the default system encoding is the only suggestion.
Are those bytes like like megabytes ?
Heads up:
Unicode: It's like a dictionary of characters. Each character has a unique entry and a value which identifies it, aka code-point. A Code-point is a 4 bytes value.
An encoding (there are several), encodes this unique code-point(4 bytes)to a sequence of bytes(variable sized), so as to save space.
Eg. A utf8 encoded character can use 1/2/3/4 bytes depending on its code-point. Similarly, a utf-16 encoded character can use 2/4 bytes.
Why utf8 is so popular you might ask? Reason is backward compatiblity to ASCII, all the ascii characters encoded to utf8 shares the same "value" when encoded to ASCII, E.g. 'A' is 65 in both encodings. All ascii chars uses only 1 byte in utf8.
Why utf16, well utf8 cannot represent all the unicode chars, there are some chars that have code-points outside the range of utf8.
News to me. Which Unicode code points do you claim cannot be encoded in utf8, then?
@@benhetland576
Just checked and it's looks like utf8 does support all code points according to wikipedia, but I'm not sure if it's correct. I saw this warning when I was working with powershell script, it needed input encoded specifically in utf-16LE since it mentioned that this supports 'all' code points. Again, not sure why it might say that.
Maybe I should make a video not just on str vs bytes, but Unicode specifically. There are lots of interesting (and dark) corners in there!
@@tusharsnn The encoding that utf-8 uses theoretically allows a max of 7 octets (or bytes if you like), in which case the first octet would start with 7 ones followed by a 0, i.e, 0xFE. The next 6 octets each encode 6 bits for a total of 36 bits, and only 21 bits are needed to cover all possible Unicode codepoints (17 "planes" of 65536 codepoints each). 4 octets encode (8 - 5) + 3 × 6 = 21 bits, so that is the longest octet sequence ever needed to encode a single Unicode codepoint. There are byte sequences that are not valid utf-8 for several reasons (even single octets like a 0xFF), but not vice versa.
There are also characters with multiple code points, such as latin V and roman numeral V, and characters with distinct glyphs such as Han unified ones, and combined characters that may be separable like ä, and the classification and ordering of characters is language dependent. Text is messy. Unicode does not provide the tools needed to compare a Chinese and Japanese phrase in one text, by design. History and politics (not necessarily national) are involved.
The best integration I've ever seen.
Thanks, James.Very concise, well-informed and well-executed . I especially like the grounded references to Python 3.1x.x My inbox needs a vaccine for "Python 4 .*" titles. I might take a look at the meta on this vid to see if I can spot such effusion for my personal ytube feed.
This seems to be a pretty good way to handle it. Meanwhile in C++ I'm still trying to figure out how to work with unicode characters at all..
Thank you! I' ve always struggled to understand the diff between str, bytes and what is encoding. And now I finally understand! Thank you 😊
So how does a sequence of four bytes get interpreted as a smiley face?
When you ask python to decode using utf8 (whether you specify the encoding specifically or whether python just does that by default), you are asking python to interpret the bytes according to the unicode standard, and the unicode standard specifies that that exact sequence of bytes means "smiling face" or some other similar description. It is then up to the author of the font you are using to create a glyph (the picture of an actual smiley) to draw whenever you try to display the smiley as a character. This allows for different sets of smileys, like the Apple smileys vs the Microsoft smileys, choosing between them is simar to choosing a different font.
Yesterday I wrote a self-extracting installer script. Today I see this.
I found it easiest to write the installer file as a string and then write the files it contains as bytes that are encoded as utf-8, especially if they are binary files. Writing the whole installer to file as bytes caused me trouble with the python interpreter.
Thanks for this.
Excellent video, thank you.
I've had a couple of issues where I've had to use the IO and locale libraries to "fix" encoding shenanigans, but I think if I revisited those lines I'd now have an actual understanding of what was happening, how the changes worked and, most importantly, how /to do it better/.
Thank you for your kind words! I'm glad this was able to help you understand and you will know how to fix it when it crops up again.
Muchas gracias, me ayudaste un montón!!!
De nada!
와 고마워요. 최고 ㅋㅋㅋ
1:21 which IDE?
always great vids. thanks a lot!
Glad you enjoyed!
Really comprehensive explanation
Great! So thankful, it cleared my confusion (I still didn’t go to Stack overflow for it 😅😂)
... May I just have curious question for an end? What might be the reason for anyone to have system not having UTF-8 as a default? (why not?)
Legacy reasons.
Before Unicode--indeed, for a long while after--there were these things called “national character sets”. In fact, there is likely still a large collection of text stored in these legacy encodings.
Nice!
Always there to teach me life ahah, thanks!
Any time!
What a coincidence! I just tried to survey the difference between them and found this video that just came out! 😀
So... why isn't your computer in UTF-8? 🤣
Ask Billy Jeans
That was the best message from a sponsor I've ever seen.
UTF8 ≠ Unicode
I was expecting you to explain the difference, since it's super related to bytes vs. str, but instead you said these terms commonly refer to the same thing 😓
FWIW here's the difference:
Unicode usually refers to the conversion between characters and numbers (ord/chr in python), and UTF8 is the conversion between these numbers and bytes.
(although Unicode is technically the name for the entire standard, including both UTF8 and the mapping between characters and numbers)
I completely agree that UTF-8 != Unicode. However, it is widespread and commonplace for developers (and others) to colloquially use the word "Unicode" interchangeably with UTF-8 as most people don't bother with technical distinctions and it is usually clear from context which conversion is meant.
Very interesting, thank you
New subscriber here. Just wanted to say that I love your videos. Very informative and fun to watch! Keep up the good work
Thanks for your kind words! Welcome to the channel and I hope you learn a lot!
@@mCoding thanks man! See you around
strife horde 🤙🤙🤙
Erisian expedition!?
@@lawrencedoliveiro9104 discord gang but 🤵♀
Out-of-tune ensemble!?
so insightful for me as a beginner!
Great video as always, thanks! But still not really clear when to use each of these types.
Bytes are mostly just useful if you mean to interpret them as something very specific that's not text. For example encryption keys or raw image data. If you're writing or handling text you should go with strings.
Fantastic explanation!
discord gang 🤙🤙🤙🤙
Always appreciated!
By default, we should be good to know why we use string in Python. But, is there a practical use case for us to use bytes ? That may helps us to differentiate the uses between string and bytes here.
many low level libraries will make you to use bytes a good example: try to write a web server from scratch
It's simply one level less of abstraction. Bytes hold arbitrary data and can be used in I/O, like storing or transmitting. String is for when data is text, while other formats could be handled with e.g. struct or ctypes. This video had a couple of examples decoding some bytes as little or big endian integers. It wouldn't make sense to pass a string to zlib.decompress() for instance.
For example, the struct module lets you convert between various Python numeric/string types and strings of bytes.
Why isn't your machine's default utf-8 if it's so popular?
Because UTF-16 is older than UTF-8, and Windows decided on the UTF-16 standard, back in the day.
And Microsoft is VERY big on backwards compatibility, which means they won't replace UTF-16 with UTF-8, unless they find a way to stay compatible.
UTF-16 is an unfortunate hangover from the early days of Unicode. Nobody uses it voluntarily any more.
Great video😊
Is there a command to tell you the system default encoding?
On Linux you would set the LOCALE to language.utf-8 but no clue if you can do that on windows
python3 -c "import sys; print(sys.getdefaultencoding())"
YAY new video!
Can you make a video on sets. How they are implemented? What is the time/space complexity of adding elements, creation from list, removing elements and others? Why it is at all in the language? Is using lists faster in specific situatutions?
Looks interesting
Small hint: Instead of writing `encoding="utf-8"`, you can just write `encoding="u8"`, which saves you up to 3 keystrokes! Check the Python doc and you can see that `u8` is just another name for `utf-8`.
Nice tip! I think i still prefer writing out the long form just for readability, but I didn't know about this shortcut before!
Just don't use u8 instead of utf-8 and then go on about your day writting for i in range(len(x)) 😁
even i knew it already since i literaly had to google it a hundret times because i found it funny to even ask the question ^^ like bites are just a string of 1 and 0 just that every 1 and 0 of your string takes at least 8 1´s and 0´s and thus is at least 8 times your memory used utf16 utf32. Let me tell you if came to the idea just useing str ('10010011' ) work with actual biits since you more comfortable with them then bytearrays dont its so painfuly slow i know cause i tryed it -.-
Make JS videos now
JS better
JS is trash 🤮
@@anon8510c better
Consider why JavaScript (and PHP) need a “===” operator, and Python doesn’t.
(Sorry, C++, this isn’t your league.)
JS is an unfortunate existence that I wish to alter.