Python's sharpest corner is ... plus equals? (+=)
ฝัง
- เผยแพร่เมื่อ 3 ก.ค. 2024
- If you thought x+=y means x=x+y you'd be wrong!
Python's plus equals augmented assignment syntax is more complex than many people realize. Let's take a look.
(Sorry for the dark purple again! I've changed it in my ide to pink for the future!)
― mCoding with James Murphy (mcoding.io)
Source code: github.com/mCodingLLC/VideosS...
Docs for iadd: docs.python.org/3/reference/d...
SUPPORT ME ⭐
---------------------------------------------------
Patreon: / mcoding
Paypal: www.paypal.com/donate/?hosted...
Other donations: mcoding.io/donate
BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------
Discord: / discord
Github: github.com/mCodingLLC/
Reddit: / mcoding
Facebook: / james.mcoding - วิทยาศาสตร์และเทคโนโลยี
More broadly, the choice in Python not to have a prominent division between things that are pointers (still can be changed when you send them off somewhere else) and things that are not pointers (are copies when they're elsewhere) is a very easy place to get tripped up, especially for those who are used to stricter languages.
Definitely my least favorite thing about python
i believe everything is a pointer in python, and nothing acts like a copy, making it more easy to understand the behavior of a program… In the last example given, L was indeed a pointer, but the += reassigned the value of the pointer to a new place, and therefore, the change couldn’t be seen from the main function… The problem comes from the fact that, if not well documented, it is difficult to know what will be the behavior of basic functions (addition, +=, …) on objects.
@@JohnChaussard While yes, everything in python is an object (and thus, a pointer), the immutability of some things make them behave more like values. But as you are saying, when specific operations are ambiguous over whether they operate on the pointer or the object, that's confusing at best (and bad language design at worst). This is all over python (see descriptors for example), and while it does enable some dark magic, I'd personally argue it does not make the language more usable.
I don't feel bad for having a severe distrust of += in python now, nice to have a clearer understanding of it though.
+= is fine, just always treat everything like a primitive and you'll be safe. In his example, if he would have returned the l var ([4,5,6]), he would have gotten the expected results.
Yess
@@SquidBeats there is no spoon
@@SquidBeats k
@@SquidBeats how do you know that Hell is real? have you seen it, if so how can I see it?
I’ve been writing Python code for a long time.. and a lot for my own videos, but I learn something new with each of these lessons! I think I need to hunker down and make sure I watch all of them.
Thanks for the support !
Same here. I'm writing python for around 7 or 8 years now. And this is the first time i saw this problem. This is because i never used python in a way that will show this behaviour (because tuple is strictly immutable, and because i use list.append and list.extend instead of adding it). But i immediately got what will happen in this code. Just didn't happen to me ever.
Jesus Christ is God and is the only way. Hell is real whether you believe it or not.
There's a principal in language design known as "least astonishment". The idea is to make a feature work in the least astonishing (aka least unexpected) way. So when you create a feature, make it work in the way most developers would expect it to work. I don't think Python got the memo on this
"Slap that like button an odd number of times"
*slaps like, then dislike, then like 2 more times*
What now mCoding?
You got me.
How about "toggle the like button an odd number of times"?
@@Pacvalham But that wouls still not have the desireable effect if the person has followed the instructions an odd number of times before.
@@mCoding
while ( not like_button.is_activated() ):
toggle_like_or_dislike()
Runtime dynamic check.
@@mrpedrobraga you went full nerd
Developer1: "+= has a bug, what should we do?"
Developer2: "Just fix it"
Developer1: "We don't have time, we must do something else"
Developer2: "Tell everyone that this is a feature, not a bug"
Developer1: "... Great Idea"
Great video! Learnt a lot form this!
Exactly!
I would say this is broken semantics, more than a bug.
An operator should not have two very distinct semantics (either modify the object binded to the variable, or rebind the variable to a new object).
There should be two distinct operators / functions, for each semantic.
@@vbregier I wouldn't say it's broken, it's just different contexts: in all cases it means "add this to that", if I said I wanted to "add a character to a string" it's obvious what my intention is: take the string, append the character to the end and wríte that back to the variable. If I said I wanted to "add a number to a number" it's also obvious what I want to do. Same goes for "adding a string to a list (of strings)".
@@adrulb The semantics of an operator should not depend on the context.
Different semantics need different operators / methods.
You are not seeing it because the word "add" itself has this problem : do you mean generate a new value, or modify the value you add to ?
Mutability of objects is crucial in programming, it must be clear which objects are modified and which are not for a given operation.
@@vbregier Yes, the problem is that concatenation is not the same as addition. Not having a separate concatenation operator is one of the few flaws in Python.
Personally the "BadList" implementation makes more sense to me. For appending things to a list, we have append and extend. a += b should be effectively the same as a = a + b
in python 2, this was the implementation, and if you iteratively created a large list, it would use SEVERAL GBs of memory (think 125GB for a 500 000 element (500 KB+) list).
My friend came across this when doing a file handling program and crashed python - It can be done in less than 10 lines of code (and it is difficult to fix).
@@mintypotato6205 wouldn't it be instantly fixable with .extend() though?
@@mintypotato6205 Difficult to fix how? Did Python 2 lists not have the .append() method or something?
7:30 Rarely comes up indeed. Can't recall ever encountering this specific issue after 10+ years of programming in Python. In any case, mutability is the root cause of this and should be handled with care.
It has only come up once in my programming career!
Man, you are great, this is exactly the content I was missing on YT. Advanced stuff, straight to the point. Keep up the good job.
Thank you for the kind words!
I didn't even know about the __iadd__ method 😅.
Also, The audio is so good now.
Thank you so much, I'm doing everything I can to up the audio quality
iAdd is part of @mCoding user created class. This doesn’t mean it’s native method, correct?
@@Sirjosephcharles the dunder iadd method is a magic method, it is used to overload the += operator.
Great topic. I had a hard time figuring this out a year ago when I was working with PyTorch - it turned out it was imperative to use x = x + foo instead of x += foo in many places since they are, counterintuitively, not the same expression... This video shed some light to the matter. Thank you!
The mutable integer id really blow my mind! I learn so much with your videos. One suggestion: you could put the python docs that you are refering to on the description :) Thanks!
docs.python.org/3/reference/datamodel.html#object.__iadd__
Unfortunately, TH-cam screws up the formatting of that URL ...
This video is fantastic. I've been using Python for years and noticed some weirdness with these operators but never dug into it. Thanks for producing such high quality educational content!
You're very welcome!
Another sharp corner: function default mutable parameters. I think most of us have fallen in this bug... I mean this _feature_
You are one of the best perk teachers around the youtube block. Enjoying your vids a lot.
Thank for the praise!
Just watched the entire video and honestly, I still feel like it's even more surprising that ++ doesn't exist
"as always, slap that like button an odd number of times"
me, an intellectual: watches the video an even number of times
>:(
Very nice channel! Keep up the good work!
Ahh Python, reminds me of good old True = False, and messing around with the cached small integer objects, changing their actual ctypes value. It's definitely good for quick prototyping but there are so many quirks... Looking forward to watching more!
Good times!
Seems “True” and “False” could indeed be (re)assigned in Python 2, but they are reserved words in Python 3. Not before time ...
You're very good at presenting technical topics.
It's a gift!
Your channel is great really enjoying your videos. I like watching intermediate/ advanced videos on python.
Glad you like them!
A talk on persistent data structures would be awesome... You don't seem that much a functional programming enthusiast, but I think it might be great to discuss limitations of standard data types in relation to copying and mutability and how it can be circumvented.
Python supports many program paradigms, including functional. This would be a great idea for future!
I've been playing with pyrsistent and I have fallen in love with using evolvers instead of mutable mappings and arrays.
Always top notch stuff from you.
Dude. Do you sleep?
Your content is of impeccable quality and is just shooting into my notif box. How do you make so many freaking good videos I love your channel ahhh
The bags under my eyes you might think that I don't ever sleep! But rest assured, I get plenty! Thank you for the support
The more I learn about other languages, the more I understand why people still write in C. It's the only language to date where things never not made sense to me.
You pass references, pointers, you can modify any piece of memory anywhere in the stack, or allocate heap, it's perfectly transparent. When working with Python, that I do often use and is great for some stuff, whenever I'm working with a new function or circumstance I never know what to expect without reading the full documentation on how it works under the hood.
You know what they say: C combines the power of assembly language with the flexibility of assembly language.
These videos are great. Hope your channel gets big.
I hope so too! Thanks for your support!
Never even touched code before. Thanks for the vid!
Another great video! Keep it Up
Thanks! Will do!
2:38 Argh. This is the sort of bug you can get when you use C macros which have parameters. And yet you get bugs like that in python even if you write something which looks safe.
Yeah, the data model for __iadd__ is broken in my opinion. Instead of usually returning `self`, it shouldn't return anything. Then, instead of the += turning into `x = x.__iadd__(y)`, it should just be `x.__iadd__(y)`. The whole point of the in-place overloads is to modify yourself instead of returning a new object, as the base versions of the operators do. Not only would this help people understand the point better, it would remove your tuple bug.
yeah this really confused me at first. I think the part that got me was the reassigning the variable after doing the iadd, which other languages typically don't
The best plus equal explanation I have ever saw on yb
Great job! Thank you for your videos!
Glad you like them!
3:27 "I can't change a 1 into a 2"
this is mostly true, but with ctypes:
import ctypes
ctypes.c_longlong.from_address(id(1)+24).value = 2
print(1)
this code prints 2
until you change the value again, or restart python, 1 is 2
Another great video! Love learning these detailed but important nuances of Python.
More to come!
Great job!
Amazing content! Keep the great work!
Appreciate it!
Pretty sure I just watched an 8-min Jedi mind trick designed to increase James' subscription count. Fair enough. Subscribed because it helps James and it feels good.
I appreciate it!
This channel is a rare treasure of useful secrets we never knew. Thanks for the great content. Looking forward to more of useful videos.
Thanks for watching!
interesting to see what COULD go wrong. I think the lesson is: don't write a class that inherits from list, and keep += for basic loops, and use .extend() for lists instead
"that can get you into a lot of trouble if you're not familiar how python works" ... and that's why it's not a feature rather a bug(to me).
That was one of the first serious case that had my and Python's relationship in trouble. "Fluent Python" also demonstrates this type of example and it had my mind blown away. You cannot just report an error on something and then do that exact thing anyways.
Other than that, great video! I don't think i've seen this "bug" covered elsewhere. It'd be cool if you covered data-model more in depth. Thanks!
Well, feature or bug, it's not that big a deal once you understand!
Saw this in a pycon video the other day. Super interesting!
Oh no! someone else covered it first!
Super interesting information!
Glad it was helpful!
I honestly didn't know the += operator can be used on anything other than arithmetic data types in python. However, is it really more convenient and clear compared to the append and extend methods for lists for the sake of argument? Nevertheless it's good of you to point this out. Thanks.
You can use any operator on any object as long as the method called by the operator is defined for the object. This is just the fundamental concept embraced by the object-oriented programming paradigm.
Create a new class with the __getitem__() dunder method defined and suddenly you can use the indexing operator on any object instantiated by your new class, and it will execute whatever you defined it to execute, regardless of whether it has anything to do with indexing at all.
Most of these "gotcha"-level videos revealing new information have less to do with actually introducing new information and more to do with just exposing our laziness in applying already understood knowledge. We all know operations in Python are in-place or not, so understanding whether the augmented operators are in-place operators is just a consequence of whether we bothered to read the documentation regarding them.
Whoah, this was such a cool video!
This channel is literally on fire
Woo!
Now I'm going to use "+=" with try except block when i want to assign a tuple member.
Thank you mCoding now i know how to use this "feature".
😵 please no
@@mCoding xD
When I first started coding in python coming from c++, I was baffled by the fact that there was no '++' to augment an integer and that I had to use the '+='. How naive I was back then! But even though I understand now that the pythonic way of doing things, rarely if ever need an index for a loop, I did not understand the specific reason why It is that way.
Thank you for this amazing explanation!
Indeed! I pretty much never plus equals for integers.
@@mCoding you should always use += where it makes sense (note that for lists etc. they have an extend method which is usually better) as it allows the implementation to optimise the operator, which is also the reason for the behaviour explained here. += is always the right choice, except when there's an appropriate method
I would be interested in hearing your views about Pythons behaviour when returning from a finally block (how it implicitly discards any unhandled exceptions prior to the finally block)
That would be a good video topic. Generally, you probably don't want to do things that pep8 unconditionally discourages :)
"It's not a bug it's a feature. *wink wink*
But sir, whould we-
*points gun* I SAID ITS A FEATURE WIN WINK
Now I have even more reasons to learn functional programming
And I have even more reason to hate Python! :D
All languages have their ups and downs!
luckily, Python has a few very neat features that allow for pure functional programs, like lambda, yield and defs inside defs... :)
can't image programming without those nowadays
@@johannestafelmaier616 Wait, what does yield have to do with pure fp?
@@mCoding no doubt. Python and I don’t get along well though I have to use it for some parts of my work. I can see why some people really like it and I do acknowledge its strengths. But it’s just really not my preference to work with. I jokingly tell my colleagues I’d rather write pure assembly but it’s all tongue and cheek and I respect Guido and his work with python as well as everyone else involved and of course everyone who’s good at programming in python. It’s just preference and my preference is elsewhere.
3:40 Interesting that the id is different every time. I presume it is derived from the memory location, similar to the default hashCode implementation in Java?
Love your content!
I appreciate that!
Please keep making content like this.
Will do!
I really really like your videos!!!
Thanks for your support!
"Make sure the thumbs up has changed color once and only once before you leave." Anyone got a problem with that?
What about colorblind people?
If you use an integer which is sufficiently maybe the reference will not change. From my own experiements I've seen that python shares reference for small numbers, presumably because they are used very often and will be a benefit to performance.
here I was thinking a+=b was just syntax sugar for a = a+b
Not a bad assumption! This is what it means in some languages for primitive types!
Quite interesting, nice video :)
Thanks, glad you enjoyed!
Wow I was running into a "bug" because of this feature and didn't know this was the cause. Thank you and thank TH-cam recommendation for popping this out on my feed.
I didn't knew that. Thanks! 😊
You bet!
I already kinda knew this inconsistency because coming from other languages you'd expect
x += y
to be short hand for
x = x + y
Which as you explained isn't true for lists.
If it were just a short hand expression it would completely avoid this Problem and python could just add another operator for in-place addition to make up for the missing functionality.
But yeah, nice video and i learned something new
Thanks!
Bro, why did you cook my brain like that? I only understood about 20% of what you were saying.
great vid!
Thank you so much!
I've almost never used it, only for increasing numerical variables, but the documentation that I check out when learning always had alternative methods that it's suggested to use
This is the kind of unexpected behavior that will have you stuck on a bug for days.
this blew my mind
Another sharp corner only slightly touched here is that .append() would do flawlessly on a list item in a tuple, because it preserves the constant pointer.
Operator overloading is quite tricky for maintaining clarity and correctness whether it's C++, Python, Haskell ...
I think the nature of overloading "hiding" function calls is what partly inspired the Zig language from the perspective of C++
How great it is to MASTER BASICS at this lvl. !!
A strong foundation is incredibly important!
this confirms my intuitive assumption that every sensible language implements "+=" just as a syntactic sugar for "var = var + value"
Fascinating, once again. You are an excellent teacher. I think the += behavior in python is highly questionable lol.
Here before this channel blows up!
Thank you so much for your support!
- Something is broken, can you fix the bug?
Python programmer: it's not a bug, it's a feature!
Finally, someone using +=
Let's stick to .append shall we. Btw is there a difference between += or .append for a list?
Yes, because += acts like extend, not append. Otherwise, not to my knowledge.
@@NateROCKS112 oh okay thanks I didn't even know += was a thing for lists tbh I just assumed
Append, extend, it's all the same for one element!
@@georgeskhater487 += is cooler because you can use it in one liners :P
@@simonconrad4402 ...extend can be used everywhere += can and more?
I have always been suspect of shorthand expressions. x = x + 1 is how I learned when BASIC had line numbers, C/C++ came along and made me hang a semicolon at the end of it, and Python circumcised it off for me again. x = x + 1 is more readable.
Incredible! Always read that tuples were immutable and they are, but no so much
The rabbit hole goes even deeper than I mentioned in this video…
So how should a custom class implement __iadd__ to not change the object itself? In other words, how should BadList be changed to behave like list, where __iadd__ works regardless of scope.
iadd is supposed to modify self (e.g use self.extend(other)) and return self if possible! This is already what the builtin list does, thankfully
This is so good.
Thanks!
These videos are great even for experienced coders, great work!
Glad to hear that!
Immutable integer types seems ridiculously inefficient, I've never understood such design decision. Ideally mutable and immutable types are available and the programmer chooses the appropriate one. In cryptography code with lots of math on big integers for example, mutable variants are far better performing unless you need a copy returned and then you want immutable. Why cant we have the best of both at our disposal.
numpy.
Vietnam/C++ copy move constructor flashbacks from this video)
Really nice knowledge about python
Haha, we've all been there!
With the benefit of bindsight, there probably should have been two different dunder methods.
* One for when the return should be reassigned to the left side of the +=
* That returns nothing (except for possibly NotImplemented). The change would be expected to happen to the first parameter (usually 'self')
That way, when trying to do += on members or elements, Python would know whether to do the final reassignment based on which is present (or which one doesn't return NotImplemented).
Of course this raises the issue of what to do if both are present, or which order they should be attempted in. So it would be a new sharp corner about _implementing_ +=
But that moves the sharp corner to the implementer, not the user of the API. But that still would have been a worthwhile improvement.
To summarize, for mutable variable m1, "+=" will create a new mutable variable m2, and then copy the content of m1 into m2.
Just remember "+=" will always create a new object and that's it
Sorry, just ignore this comment, what I sad is wrong.
Me: *doesn't use python at all since I use C++ on more occasions*
also Me: "ooh new video from mCoding about python? time to drop my projects and watch"
I appreciate it so much!
if i not wrong, numbers from 0 to 255 has preset id and it is one of the reason of changing id there 3:30
I found all of these example completely obvious, natural and well... correct.
It's not even weird, how _else_ would they work?
...I may have used Python extensively during my formative years.
I should make a duplicate video of += in C++ and that will scar you!
@@mCoding I developed php for eight years, C++ doesn't scare me.
It feels as if they would have been better off not adding these operators to the language. They mean something clear in other languages but that doesn't fit well with the data model of Python so it would have been clearer to force the one extra keystroke of x=x+1 instead of x+=1.
Great channel! New sub. Thanks much, can't wait to binge your videos all weekend 😊👍
Use C: Make it complicated. Use Python: Add complexity.
Even I do not use Python to program with, it's enlightened to see it's pitfalls.
Pitfalls, or necessary consequences? Sometimes it's hard to tell!
@@mCoding I see them as pitfalls, because it breaks intuition.
Ive had a bad experience with this in a sorting simulator... best way i had was to return the lists from the functions
Yeah, it kind of forces you to use temporaries
Will there be more weirdness for hashable types (additional step in whole assignment procedure) where the += operator defined?
I don't think the hash will come into play, perhaps someone else has a counterexample?
This thing almost made me loose my sanity when I tried to implement it in one of my projects. Basically I wanted to make a dummy class that would do nothing on __iadd__ except memorising that __iadd__ was called. And due to other implementation details this __iadd__ ended up setting weird attribute values to other seemingly unrelated objects. It took me a while to figure out the reason
How do you come across these quirks?
I push every language to its limits and get burned!
Reminds me of the mutable default argument "gotcha." def frobnicate(foo=[]): foo.append(1); return foo; has surprising behavior when you run with the default argument more than once.
Yeah, that was surprising.
You just keep attending to the same list.
best to handle that I know this is
def foo(l=None):
l = l or []
...
Does this also apply to the non in-place += operator? I.e. x = x + 1. I assume not, since iadd is not called there.
Actually, x=x+1 does something completely different! Furthermore, a raw plus operation is expected to make a copy, not modify the original, and also, a raw equals sign is expected to rebind the variable.
Python is built on C. Arrays always have the first element as the pointer, so as you increase the list size the memory address stays the same. Integers don't tend to do this in C. The value is either written into a new memory location or it uses an enumerated one.
CPython is built on C. Python itself is an abstract language, with no pointers or adresses.
"Slap that like button an odd number of times!"
I like that lol
Ty Ty!
Nice video
tron bad
Thanks!
I did learn that, almost 30 years ago, but it was on C++