It took me so long to understand covariance and contravariance of type variables, and honestly I would love to see you explain it, just so more people understand how powerful of a concept it really is
I always use the helper analogy of: Covariant means something of that type can "come out" (via return value), contravariant means it can "contract" something of that type (via parameter e.g.). Another more technical way is that the less confusing sounding name (covariance) just preserves inheritance, while the more confusing one (contravariance) inverts it fully.
Covariance and contravariance are useful when you need mutable mappings, which most of the time is not the case. It's sad that we don't have an immutable dict type where we don't have to deal with these complexities. Also hashing would be nice
After surveying many python instruction channels on youtube I can confidently sayths is the best channel to learn about interesting facets of python programming. Your content should be sponsored by somebody like Khan Academy or LeetCode. I am baffled that you do not have more subscribers. Thank you for all of your efforts!
I've actually had a situations where an InvertibleDict would've been useful. And I'm glad typehints are optional. Great for when you're building something bigger or sharing code with others. But if I just want to get some data analysis code, it's great to not have to worry about all those intricacies. #pycharm
It happens more frequently than one would think at first glance. Allowing the use of one shortened or abbreviated version of an identifier is quite common and and expected convenience feature.
Knowing about this invertible dictionary stuff would have been tremendously useful to me on pretty much the first python project I undertook on my own. I was building a simulator for the Enigma machine from WW2. Spent more than half a day trying to figure out how to make the forward and backward dictionaries of the Enigma's rotors behave correctly when the rotor steps before encrypting each character of the plaintext. Ended up throwing in the towel and just generating the backwards dictionary on the fly when it was needed using a dictionary comprehension. Although using this InvertibleDict approach would have made my code longer, so maybe what I came up with ended up being better.
I recently came across the exact situation you describe and found it quite unsatisfying to manually define two dicts that are inverses of each other. This is very elegant, thanks a lot! #pycharm
Your explanation of ABC's was very good, I didn't know you could register classes to be counted as instances of ABC's. That makes me interested in how that's accomplished under the hood. I look forward to your explanation of covarient and contravarient types. #pycharm
One downside is that inheriting from an ABC changes the metaclass to ABCMeta, which may conflict with other metaclass stuff you (or your users) are doing, or you may just want to avoid metaclasses. Additionally, many of collections ABCs use multiple inheritance (Collection inherits from Sized, Iterable, and Container), which is another thing that some people prefer to avoid.
I've been programming for a while and only this month had a reason to look up what covariance is for typing, it's pretty neat but I definitely understand why you glossed over that :P In general I prefer interfaces to this direct inheritance as many languages only support one parent class, but as Python is Python I do appreciate what they offer with the ABCs
Great video: Nice summary of the basic concept, sprinkled with interesting sidenotes (e.g. i didnt know about the convention for hidden arguments)! Deliberately won't write the hashtag to avoid lowering the chance for people who would make better use of the license than me. 😊
In your implementation, the constructor always copies reference to the forward argument, even if backward isn't given. This can lead to the strange behavior when you create InversibleDict from other mapping and mutate one of them. Furthermore, if the original mapping was immutable, you'll get an error. I suppose #pycharm won't highlight that kind of error.
I think the subtlety here is that the __init__ takes in _forward by-value, not by reference. In rust, this is very explicit. C++ also has an explicit way to specify this. I suppose it is a feature we cannot easily get in python however. The implementation and type hinting for __init__ is technically correct (sound), but it is misleading. So I guess it is not actually correct in practice.
Hey that's a good point! In this example I made InvertibleDict assume ownership of the dict you pass in. Instead, you could make a copy of the forward dict if the backward one wasn't provided.
@@coarse_snad all classes in Python are copied by reference unless explicitly stated the opposite. Otherwise, the interpreter would have to handle return-value-optimisations which is incredibly hard to implement in the dynamic language. And when impossible, each stack call would almost double the memory required for the program to run.
For the interfaces commonly used in Python, it is very convenient to use the abc module. But as far as I know most of the time you only need to use Protocol to type hint duck type custom classes. #pycharm
It's fun to see you present real python programming. As an example, in a codebase i maintain there is currently a very long block comment with several examples to try to explain why a type[ignore] is required on a specific line. The block comment links to several github issues for various projects that are part of python, none of which are closed yet. A good explanation of variance is hard to come by, so i would be very interested to see your possible future video on it. It's an important subject that many languages try to ignore, sadly.
Pretty good video regarding ABCs, I have been using ABCs for quite a while now, yet this video dived a little deeper into the topic and I learnt a lot more than what I wished I would have learnt earlier! #pycharm
Hello, great video! One little thing: likely it would be more effivcient to cache the .inv method to avoid recreating the class each time one wants an inversely mapped element? Or am I missing something here? Best wishes!
Besides the interesting and great explanations, the example you give, an invertible dict looks very useful in many situations and projects. How do you organize such things? Do you create a package for each such self contained utility? Do you have a private "helpers" package that contains all such utilities? Or do you copy paste them into each and every project that uses it?
At around 5:25 you mentioned that one could be in a situation where you can't inherit from some class. I have a hard time imagining what that would be, you don't by any chance have an example of that?
There are many reasons! It could be for efficiency like if you are writing a C extension class, inheriting from a Python class could ruin it. It could be because you are using metaclasses and inheriting from an ABC would cause a TypeError because the metaclasses are different (inheriting from ABC changes the metaclass to ABCMeta, and metaclasses of derived classes must be subclasses of the metaclasses of all bases). It could be because your boss told you to use your company's internal BetterMappingInterface as the base class. It could be because you are trying to avoid multiple inheritance (Collection inherits from Sized, Iterable, and Container!).
Have you ever thought about doing Julia videos? I’m trying to get into the language but I’ve realised I learnt so much Python on here and I don’t have the same resources to teach me the idiosyncrasies and pitfalls of Julia! So I’m just floundering in the dark poring over dense documentation.
One thought here -- the pattern of a dict's init taking a dict as a parameter is that it makes a COPY of the dictionary. What you've done here makes an IterableDict that references the dictionary passed in. That might not seem like a big deal -- but you've gone to some lengths to maintain the invariant conditions here, and there's nothing to stop the caller from modifying the dictionary they gave you after the fact, breaking that. You might want to instead make a copy of a dict if an external client passes one in, but have both _forward and _backward internal parameters you can use where you can trust they're not something somebody is (in good faith) passing in normally with dict-inspired expectations that don't line up with the implementation.
An invertible dictionary is really a great idea. I had two implement a two way mapping before but I used two dicts. Your solution looks definitely cleaner. #pycharm
Great video as always! Can you make a video on function/method overload in Python? There are options such as singledispatch, multipledispatch, multimethod and even typing module has an overload decorator
Would it be possible to create an invertable dict that allows for multiple keys to have the same value? If you wanted to return the keys for a certain value it would be value: list(keys) ?
Very interesting video. I got one question: Why does changing the value in the inverse dict change the value in the "original" one? The inverse "inv" property creates a new instance and therefore instanciates the new class with copies from the "old" instance. Changing values in here shouldn't affect the "old" instance. What am I missing out?
Thanks for your answer. As far as i know python doesnt support references or pass by reference. Could you rewrite the part you are talking about for me more explixit? That would realy help.
This is because d and d inv are sharing the same underlying forward and backward dicts, just swapped. This is just like if you have a=d and b=d, then changing a[1] is also changing b[1].
@@mCoding Thank you for the answer. I tried it in a sample code and its very interesting. E.g. When I pass a list to a function and change that list in the function without returning anything, the list in the main scope changes. too I always thought the arguments get copied and don't affect the main scope. Seems like a way for functions to return values (append to a list defined in the main scope and passed as argument) without the need of an explicit return statement. Very interesting.
Great video! I recently used similar functionality via bidict library in my commission, where I mapped ids to user search queries for fast bidirectional look-up. Pretty useful tool And as fun-fact about #pycharm, did you know that the IDE marks type of monkey-patched function arguments as name of functions it used? In my case it's been "acurs: {execute, fetchall} = None". I discovered it when refactoring the boilerplate code of database connections, as I used context managers for auto-commiting and closing them. With a simple decorator factory and an extra argument for functions using it, I could avoid 2 extra indentation levels and get full access to the DB - and the code looks so much better 👍
This is an interesting dive into the abc module. I only recently started using aspects of abc (I'm using abstractmethod for a class structure), and this video made me want to implement complex interfaces into my own code, LOL. Great video as always! #pycharm
Maybe it's just me, but it sounded like agile methodology is something evil. As a person who worked in a dev team for a business products (yes, not the same as a video game, but hear me out), who switched from waterfall to agile, I can say that agile is not about releasing anything in two weeks no matter the quality. It's about delivering to the customers "value" in a short time, instead of making them wait longer and deliver to them something they already don't need. As a member of a team you benefit, if there is an error in the requirements you notice it faster, qa test smaller pieces of functionality instead of whole thing with a lot of subsystem and integrations at once. In this line of business, I find agile a huge help. But it's just a tool. Does this tool fit the game industry? I haven't worked in gamedev, but I think it depends. Online games for example. For sure it fits. Big single player experiences, I think it's possible, if you do it right. Don't deliver each sprint to the players, deliver it to "stakeholders" and focus groups, you'll be able to keep most of the benefits (if not all) of agile, higher-ups would always know if the project is going as planned or if there are any hiccups (being open is a part of agile), and then release it when all the parts are done. I think you can find something similar in early access indie games, but they actually release "value" then it's ready, not at the and of the project.
This is awesome, I have a use case where I have separate databases where I want to map one table's primary keys to the other table's primary keys. #pycharm
I really like your explanations and videos. They make me want to get deeper into topics and learn more. Of course I would love to have a chance to get those juicy #pycharm redeemable licence codes. Keep up the good work!
Great example! I think the name BijectiveDict or BidirectionalDict would have been better names though. It instantly communicates the underlying mathematical principal
When you define the inv function, you call self.class(self._backward, _backward=self._forward). Does this instantiate a new class with ~references~ to the same _forward and _backward objects in memory, or does this copy those objects? I'm wondering if there are unneeded, repeated computations every time the inv function is called.
Just the default Darcula with a few font color changes to make them more readable on screen (e.g. made __dunder__ variables pink instead of dark purple).
I've has multiple problems with typehinting when using "set/tuple/list/dict" instead of using those from the typing library "Set/Tuple/List/Dict", is it a behavior for newer pythons? I am kinda atuck in 3.9 bc aws wont update the lambda runtimes
The lowercase ones were introduced in 3.9 so you should be ok. The uppercase ones are deprecated and may disappear in the future. Functionally they are the same, but it is recommended you transition to the lowercase ones as you upgrade your Python.
When implementing __setitem__, why do you use a paradigm of creating a dummy missing object instead of just checking if value in self._backward and self.backward[value] != key? It's one fewer O(1) check, but it seems less readable. #pycharm
Hashing an element is not necessarily O(1), it can be arbitrarily complex, which is why it's generally recommended to hash only when necessary. Although in real code with normal types it probably doesn't make any noticeable difference. I don't find the dummy missing any less readable, it's really up to taste, feel free to implement it however it makes the most sense to you. The MutableMapping ABC prefers to wrap everything in try: except KeyError: everywhere, which is another fine alternative to both your and my suggestion.
In the situation where two keys have the same value (9:30), why throw an error instead of deleting the old key/value pair from each dictionary? This is what you do in the next case. Wouldn't this also preserve the enforced bijection? This seems to help, but I can't pin down why... #pycharm
Great question! It's totally up to you how you want your map to act and I even coded that version up too. Personally I decided that d[k] = v deleting a key unrelated to k would be very confusing behavior that would make InvertibleDict not fit my mental model of how a dict should act. If you prefer it a different way, feel free to change the implementation accordingly 😉
@@mCoding Gotcha! I also suppose that throwing an error allows the user to decided when (or if) the unrelated key is deleted. It seems, as always, different implementations for different use cases. Thanks :)
For certain situations that works, particularly when your mapping type is a thin wrapper around a single actual dict, but in others it may take some "forcing" to get similar functionality. For instance, with the InvertibleDict, how would you implement inv if you inherited from dict? The inv property does not create new dicts when making the inverse InvertibleDict, it reuses the existing ones both for efficiency and so that operating on the inverse also modifies the original. However, if I inherited from dict I would necessarily be creating a new dict when I create the inverse, so instead I would probably have to create a "View" type in addition that implements the MutableMapping protocol, but that's already what InvertibleDict is.
@@mCoding I’m sorry, I should have been more elaborate: Yes, inheriting `dict` directly is often not a good solution. I meant my question more in this sense: 1) Does it make sense to inherit `dict` at all, are there any cases? 2) What problems lie in this direction? This would have made a great introduction to why the base classes exist in the first place and why inheriting them instead of the real thing is often the better choice.
1:45 I think the "I won't modify you" is wrong. Sequence, Set, and Mapping can be mutable. If you do "isinstance" with list and tuple against Sequence and MutableSequence, the only one that is False is tuple against MutableSequence.
Great observation and this is a subtle point. Type annotations are not checked by Python and so their meaning is highly dependent on who is using them. Different people can hold themselves and their team to different standards regarding how they are used and whichever standard you prefer to use is up to you. Read another way, it's often unspoken rules that dictate what they mean. Most people agree that b: B means b should be an instance of B, either literally in the sense of passing an isinstance check or structurally for ABCs and Protocols. So at the bare minimum d: Mapping means d should be a Mapping. As you point out, being a Mapping doesn't mean you aren't also a MutableMapping, so code that takes your Mapping, checks to see if it is a MutableMapping using isinstance, and then mutates it is technically allowed because you checked the type. However, there's a stricter level of interpretation for d: Mapping, that you promise to program only against the Mapping *interface*. If you hold yourself to the standard of programming against the Mapping interface, checking to see if your argument is a MutableMapping and then mutating it would certainly be unwelcome behavior :), even though it is not going to error at runtime. So a common unspoken rule here is that if you type hinted d : Mapping, you aren't going to call any mutating methods even if they exist, and that if you really wanted to mutate the mapping, you would have typed it as MutableMapping. Again, this is pure convention and many people do not abide by this convention, but that convention is what I was tacitly using here when I said "I won't modify you". It's not that I can't modify you, it's that if I intended to call any modifying methods I would have used MutableMapping.
Great question! The reason is because the user may have intended to store the value None in the dictionary, e.g. {"a": None} or {None: "a"}. We need to be able to distinguish this case from the key/value not previously existing.
@@mCoding Aha, hadn't thought of that. But in that case, wouldn't it be more clear if we explicitly check if the key exists, instead of providing a default to get, and checking if we got the default?
The inv property is a little expensive though. It creates a new object every time it is invoked. If the dict has too many items, the dict comprehension used in __init__ can be a little expensive. I don't have a better solution though. #pycharm
@@tylerfusco7495 maybe. But you would have to refresh the cache either time the forward or the backward dictionary is updated. The net effect depends on the ratio of reads to writes.
@@Mayur7Garg you would not need to refresh it, it would always be up to date, as there is no way to swap the underlying dicts (within the public interface). Everything is a reference, and the dicts are not copied for the inv property. I would also argue that the overhead of creating a single python object, especially when slots are being used, is silly to worry about. Python will have larger overheads everywhere, so optimizing this part is not an effective solution.
If you really want to avoid the overhead of the extra class creation, i would add an extra field to the slots specifically for the cached inverse mapping. You would probably need to add extra complexity to __init__, but it would allow you to simply return the cached counterpart in the inv property. This would mean that for an InvertibleDict a, "a.inv.inv is a" returns True.
@@coarse_snad Yeah.. I was actually looking at the dict comprehension call but now I realised that won't be called during inverting. During the inversion, only references need to be updated since the backward mapping is not None.
3:08 "How restrictive this becomes for the caller of the function" I have never heard of python ever enforcing type hinting. I could have "def foo(x: float): pass" and call with "foo(2.0)" or "foo([1, 2])" and get away with it. I've only ever heard of type hinting applying with LINTERS, not runtime. Sure, someone reading the library code would go, "Oh, this function is expecting a float, not a list". Sure, but like pirates of caribbean, they're more like guidelines.
The types are indeed suggestions, for now, but they may not be in the future. Generally it's a good idea, and good practice, to listen to yourself if you suggest that something may be a specific type.
You're right that it is not the python runtime that complains, it is the static analysis tools like linters, IDEs, and type checkers. In the professional world, these static analysis tools are very much a part of everyday life and depending on the project, failing a type check in CI means failing the build. Additionally, regardless of whether this is an actual blocker, type hints are a form of communication of intent between programmers. If i type hint a function as taking an int, you should not pass a str to it. If i type hint a function as taking a dict, you should not pass a dict-like thing that doesnt inherit from dict. So whether this is enforced by your CI system or not, other programmers will be affected by restrictive type hints.
@@mCoding Just be more careful about your wording in the next video. I was not saying anything about the merits of type hinting. The wording in your video made it seem like the langue enforces types: "How restrictive this becomes for the caller of the function". Maybe it was not clear as to how the caller is "restricted". Is in the human author? Is it the python run time engine? What is the "caller"? What is the "restriction"? I live in a world where everyone seems to know what everyone else means, like an inside joke or whatever. Tear me apart for why I'm responsible for misunderstanding you, but at the end of the day I still lose out. Just please, next time speak more clearly. Thank you.
Experimental chrome flag, although as it was kindly pointed out to me, the docs for Python 3.12 allow you to select your theme, and it looks much nicer.
It took me so long to understand covariance and contravariance of type variables, and honestly I would love to see you explain it, just so more people understand how powerful of a concept it really is
this is not python, rather rust. still a very useful video, and helped me a lot: th-cam.com/video/iVYWDIW71jk/w-d-xo.html
I don’t fully understand type hinting beyond basic usages. Feel so anticipated too see one tutorial about them from mcoding!!
I always use the helper analogy of: Covariant means something of that type can "come out" (via return value), contravariant means it can "contract" something of that type (via parameter e.g.).
Another more technical way is that the less confusing sounding name (covariance) just preserves inheritance, while the more confusing one (contravariance) inverts it fully.
Covariance and contravariance are useful when you need mutable mappings, which most of the time is not the case. It's sad that we don't have an immutable dict type where we don't have to deal with these complexities. Also hashing would be nice
After surveying many python instruction channels on youtube I can confidently sayths is the best channel to learn about interesting facets of python programming. Your content should be sponsored by somebody like Khan Academy or LeetCode. I am baffled that you do not have more subscribers. Thank you for all of your efforts!
Thank you very much, I appreciate it!
Excited for the 15 parts series j/k. Would love to see a discussion of covariance and contravariance for python, ty! Love #pycharm
Every time I see you have posted, I just know there will be a high quality video on an advanced topic. Love that.
I've actually had a situations where an InvertibleDict would've been useful. And I'm glad typehints are optional. Great for when you're building something bigger or sharing code with others. But if I just want to get some data analysis code, it's great to not have to worry about all those intricacies. #pycharm
It happens more frequently than one would think at first glance.
Allowing the use of one shortened or abbreviated version of an identifier is quite common and and expected convenience feature.
Knowing about this invertible dictionary stuff would have been tremendously useful to me on pretty much the first python project I undertook on my own. I was building a simulator for the Enigma machine from WW2. Spent more than half a day trying to figure out how to make the forward and backward dictionaries of the Enigma's rotors behave correctly when the rotor steps before encrypting each character of the plaintext. Ended up throwing in the towel and just generating the backwards dictionary on the fly when it was needed using a dictionary comprehension. Although using this InvertibleDict approach would have made my code longer, so maybe what I came up with ended up being better.
This channel has the best videos for serious programmers #pycharm
Actually, the part you omitted would be the most interesting to watch. I hope there will be a part 2 of this video.
The GOAT of Python has returned!
I recently came across the exact situation you describe and found it quite unsatisfying to manually define two dicts that are inverses of each other. This is very elegant, thanks a lot! #pycharm
Secret giveaway? I think you mean _giveaway
I lol'd
Your explanation of ABC's was very good, I didn't know you could register classes to be counted as instances of ABC's. That makes me interested in how that's accomplished under the hood. I look forward to your explanation of covarient and contravarient types. #pycharm
Why would you use .register, even if you implement everything yourself? Is there any downside to inheriting and overriding everything?
One downside is that inheriting from an ABC changes the metaclass to ABCMeta, which may conflict with other metaclass stuff you (or your users) are doing, or you may just want to avoid metaclasses. Additionally, many of collections ABCs use multiple inheritance (Collection inherits from Sized, Iterable, and Container), which is another thing that some people prefer to avoid.
@@mCoding Interesting, thank you very much!
I've been programming for a while and only this month had a reason to look up what covariance is for typing, it's pretty neat but I definitely understand why you glossed over that :P In general I prefer interfaces to this direct inheritance as many languages only support one parent class, but as Python is Python I do appreciate what they offer with the ABCs
'Hey, kids! Today we're going to learn the ABC! That's right, little TImmy, Abstract Base Classed!!'
Timmy didn't realize his life was set down a different path that day.
It's a shame you joked about making this a 15-part series, I would definitely watch all of it ;) #pycharm
thanks again for keeping up with the intermediate-advanced Python videos. much appreciated. #pycharm
Great video: Nice summary of the basic concept, sprinkled with interesting sidenotes (e.g. i didnt know about the convention for hidden arguments)! Deliberately won't write the hashtag to avoid lowering the chance for people who would make better use of the license than me. 😊
People go crazy with type hinting. But seeing how #pycharm can use it to infer autocomplete suggestions is pretty slick!
In your implementation, the constructor always copies reference to the forward argument, even if backward isn't given. This can lead to the strange behavior when you create InversibleDict from other mapping and mutate one of them. Furthermore, if the original mapping was immutable, you'll get an error. I suppose #pycharm won't highlight that kind of error.
I think the subtlety here is that the __init__ takes in _forward by-value, not by reference. In rust, this is very explicit. C++ also has an explicit way to specify this. I suppose it is a feature we cannot easily get in python however.
The implementation and type hinting for __init__ is technically correct (sound), but it is misleading. So I guess it is not actually correct in practice.
Hey that's a good point! In this example I made InvertibleDict assume ownership of the dict you pass in. Instead, you could make a copy of the forward dict if the backward one wasn't provided.
@@coarse_snad all classes in Python are copied by reference unless explicitly stated the opposite. Otherwise, the interpreter would have to handle return-value-optimisations which is incredibly hard to implement in the dynamic language. And when impossible, each stack call would almost double the memory required for the program to run.
For the interfaces commonly used in Python, it is very convenient to use the abc module. But as far as I know most of the time you only need to use Protocol to type hint duck type custom classes. #pycharm
A video on the typing module and some advanced use cases? Like annotating a mixin class?
It's fun to see you present real python programming.
As an example, in a codebase i maintain there is currently a very long block comment with several examples to try to explain why a type[ignore] is required on a specific line. The block comment links to several github issues for various projects that are part of python, none of which are closed yet.
A good explanation of variance is hard to come by, so i would be very interested to see your possible future video on it. It's an important subject that many languages try to ignore, sadly.
These are more and more advanced topics. I'm so glad, keep it up! #pycharm
Pretty good video regarding ABCs, I have been using ABCs for quite a while now, yet this video dived a little deeper into the topic and I learnt a lot more than what I wished I would have learnt earlier! #pycharm
Love your videos! Watching them whole, so couldn't miss the note about secret #pycharm contest ;)
Typehinting and strict data thpe restriction are something really useful for any level of collaboration of project.
Hello, great video! One little thing: likely it would be more effivcient to cache the .inv method to avoid recreating the class each time one wants an inversely mapped element? Or am I missing something here? Best wishes!
Thanks for the great chanel and content #pycharm
This was a fantastic sample use case; big brain, very instructive #pycharm
It's pretty much Rust traits!
Complete with default implementations!
Very cool!
The student becomes the teacher. Good to see languages develop together :D
Can't wait for the other 14 parts! :D :D
Very interesting video, as always! Always learning a lot from you on this channel! #pycharm
Besides the interesting and great explanations, the example you give, an invertible dict looks very useful in many situations and projects. How do you organize such things? Do you create a package for each such self contained utility? Do you have a private "helpers" package that contains all such utilities? Or do you copy paste them into each and every project that uses it?
Typically, you would publish the utility as a package on either pypi or your company's private internal pypi server.
At around 5:25 you mentioned that one could be in a situation where you can't inherit from some class. I have a hard time imagining what that would be, you don't by any chance have an example of that?
There are many reasons! It could be for efficiency like if you are writing a C extension class, inheriting from a Python class could ruin it. It could be because you are using metaclasses and inheriting from an ABC would cause a TypeError because the metaclasses are different (inheriting from ABC changes the metaclass to ABCMeta, and metaclasses of derived classes must be subclasses of the metaclasses of all bases). It could be because your boss told you to use your company's internal BetterMappingInterface as the base class. It could be because you are trying to avoid multiple inheritance (Collection inherits from Sized, Iterable, and Container!).
I would love to see a video on covariant and contravariant and how it may affect autocomplete in #pycharm
Have you ever thought about doing Julia videos? I’m trying to get into the language but I’ve realised I learnt so much Python on here and I don’t have the same resources to teach me the idiosyncrasies and pitfalls of Julia! So I’m just floundering in the dark poring over dense documentation.
#pycharm I realy like how you have taught me more on how to write better code.
One thought here -- the pattern of a dict's init taking a dict as a parameter is that it makes a COPY of the dictionary. What you've done here makes an IterableDict that references the dictionary passed in. That might not seem like a big deal -- but you've gone to some lengths to maintain the invariant conditions here, and there's nothing to stop the caller from modifying the dictionary they gave you after the fact, breaking that.
You might want to instead make a copy of a dict if an external client passes one in, but have both _forward and _backward internal parameters you can use where you can trust they're not something somebody is (in good faith) passing in normally with dict-inspired expectations that don't line up with the implementation.
Great point! If i ever pull this out into a repo, i'll be sure to be more careful about copy/ref.
Great video as always! #pycharm
An invertible dictionary is really a great idea. I had two implement a two way mapping before but I used two dicts. Your solution looks definitely cleaner. #pycharm
I mean... he also uses 2 dicts here...
Thanks for the great content on advanced topics. #pycharm
Great content as always. Would love to hear your pov on mojo and codon #pycharm
So happy to have discovered your channel #pycharm
Once again another useful video!!
#pycharm
That video is interesting, despite not being the level of Python I need daily #pycharm
#pycharm i would watch 15 part of this :)
Just starting binging all your videos #pycharm
Great video as always! Can you make a video on function/method overload in Python? There are options such as singledispatch, multipledispatch, multimethod and even typing module has an overload decorator
Would it be possible to create an invertable dict that allows for multiple keys to have the same value? If you wanted to return the keys for a certain value it would be value: list(keys) ?
Very interesting video. I got one question: Why does changing the value in the inverse dict change the value in the "original" one? The inverse "inv" property creates a new instance and therefore instanciates the new class with copies from the "old" instance. Changing values in here shouldn't affect the "old" instance. What am I missing out?
It's because he's using a reference. It'd be easier to understand if more explicit syntax was used.
Thanks for your answer. As far as i know python doesnt support references or pass by reference. Could you rewrite the part you are talking about for me more explixit? That would realy help.
This is because d and d inv are sharing the same underlying forward and backward dicts, just swapped. This is just like if you have a=d and b=d, then changing a[1] is also changing b[1].
@@mCoding Thank you for the answer. I tried it in a sample code and its very interesting. E.g. When I pass a list to a function and change that list in the function without returning anything, the list in the main scope changes. too I always thought the arguments get copied and don't affect the main scope. Seems like a way for functions to return values (append to a list defined in the main scope and passed as argument) without the need of an explicit return statement. Very interesting.
Great video!
I recently used similar functionality via bidict library in my commission, where I mapped ids to user search queries for fast bidirectional look-up. Pretty useful tool
And as fun-fact about #pycharm, did you know that the IDE marks type of monkey-patched function arguments as name of functions it used? In my case it's been "acurs: {execute, fetchall} = None".
I discovered it when refactoring the boilerplate code of database connections, as I used context managers for auto-commiting and closing them.
With a simple decorator factory and an extra argument for functions using it, I could avoid 2 extra indentation levels and get full access to the DB - and the code looks so much better 👍
This is an interesting dive into the abc module. I only recently started using aspects of abc (I'm using abstractmethod for a class structure), and this video made me want to implement complex interfaces into my own code, LOL. Great video as always! #pycharm
Maybe it's just me, but it sounded like agile methodology is something evil.
As a person who worked in a dev team for a business products (yes, not the same as a video game, but hear me out), who switched from waterfall to agile, I can say that agile is not about releasing anything in two weeks no matter the quality. It's about delivering to the customers "value" in a short time, instead of making them wait longer and deliver to them something they already don't need. As a member of a team you benefit, if there is an error in the requirements you notice it faster, qa test smaller pieces of functionality instead of whole thing with a lot of subsystem and integrations at once. In this line of business, I find agile a huge help. But it's just a tool. Does this tool fit the game industry? I haven't worked in gamedev, but I think it depends. Online games for example. For sure it fits. Big single player experiences, I think it's possible, if you do it right. Don't deliver each sprint to the players, deliver it to "stakeholders" and focus groups, you'll be able to keep most of the benefits (if not all) of agile, higher-ups would always know if the project is going as planned or if there are any hiccups (being open is a part of agile), and then release it when all the parts are done.
I think you can find something similar in early access indie games, but they actually release "value" then it's ready, not at the and of the project.
This is awesome, I have a use case where I have separate databases where I want to map one table's primary keys to the other table's primary keys. #pycharm
Thanks for the informative content! #pycharm
thanks, would be also interesting to hear about covariant/contravariant typing #pycharm
I really like your explanations and videos. They make me want to get deeper into topics and learn more. Of course I would love to have a chance to get those juicy #pycharm redeemable licence codes. Keep up the good work!
Thanks very much and luck is on your side today because you will soon be the owner of one of those juicy #pycharm license codes! Email me to claim.
Great example!
I think the name BijectiveDict or BidirectionalDict would have been better names though. It instantly communicates the underlying mathematical principal
I agree! Unfortunately, i've learned the hard way that it's not good to assume programmers know the word bijective. That's why I went with invertible.
Pleeeeas talk about covariance and contravariance :)
Very nice video as always!
To be honest I wouldn't mind programming content in TH-cam Kids, the children deserve to get some education
#pycharm
안녕
Keep up🎉
Great video, as always #pycharm
you know what i really like using #pycharm
#pycharm
Awesome video also always.
Great video! #pycharm
When you define the inv function, you call self.class(self._backward, _backward=self._forward). Does this instantiate a new class with ~references~ to the same _forward and _backward objects in memory, or does this copy those objects? I'm wondering if there are unneeded, repeated computations every time the inv function is called.
The original and the inverse share the same underlying dictionaries so the data in the dictionaries are not copied.
one of my favorite videos of yours :) great job!
early access gang 🤙
Do you usaully use a theme when you use #pycharm?
Just the default Darcula with a few font color changes to make them more readable on screen (e.g. made __dunder__ variables pink instead of dark purple).
I've has multiple problems with typehinting when using "set/tuple/list/dict" instead of using those from the typing library "Set/Tuple/List/Dict", is it a behavior for newer pythons? I am kinda atuck in 3.9 bc aws wont update the lambda runtimes
The lowercase ones were introduced in 3.9 so you should be ok. The uppercase ones are deprecated and may disappear in the future. Functionally they are the same, but it is recommended you transition to the lowercase ones as you upgrade your Python.
Any plans on a mojo video?
When implementing __setitem__, why do you use a paradigm of creating a dummy missing object instead of just checking if value in self._backward and self.backward[value] != key? It's one fewer O(1) check, but it seems less readable. #pycharm
Hashing an element is not necessarily O(1), it can be arbitrarily complex, which is why it's generally recommended to hash only when necessary. Although in real code with normal types it probably doesn't make any noticeable difference. I don't find the dummy missing any less readable, it's really up to taste, feel free to implement it however it makes the most sense to you. The MutableMapping ABC prefers to wrap everything in try: except KeyError: everywhere, which is another fine alternative to both your and my suggestion.
interesting vid, thanks. #pycharm
Great video
thanks for another really good explanation! #pycharm
You are very welcome, also you get a #pycharm license! Email me to claim.
@@mCoding 98
heck yeah #pycharm
I thought abc is just a interesting name, never expected it means Abstract Base Class. LOL #pycharm
could someone tell me what line 26 does at 7:35, please? i have never seen this before and am very intrigued
Absolutely! I've got a whole video on keyword-only and positional-only args! th-cam.com/video/R8-oAqCgHag/w-d-xo.html
Thank you for the always very informative videos =)
PS: Post a comment containing #pycharm to enter!
I'd like to have #pycharm
Can somebody explain the difference between interfaces and abstract classes in the context of python?
#pycharm
my educational license ends by the 1st of june :
Have you considered using a free editor such as vim or emacs?
In the situation where two keys have the same value (9:30), why throw an error instead of deleting the old key/value pair from each dictionary? This is what you do in the next case. Wouldn't this also preserve the enforced bijection? This seems to help, but I can't pin down why... #pycharm
Great question! It's totally up to you how you want your map to act and I even coded that version up too. Personally I decided that d[k] = v deleting a key unrelated to k would be very confusing behavior that would make InvertibleDict not fit my mental model of how a dict should act. If you prefer it a different way, feel free to change the implementation accordingly 😉
@@mCoding Gotcha! I also suppose that throwing an error allows the user to decided when (or if) the unrelated key is deleted. It seems, as always, different implementations for different use cases. Thanks :)
Thanks #pycharm
What about simply inheriting from `dict` instead of `abc.Mapping`?
For certain situations that works, particularly when your mapping type is a thin wrapper around a single actual dict, but in others it may take some "forcing" to get similar functionality. For instance, with the InvertibleDict, how would you implement inv if you inherited from dict? The inv property does not create new dicts when making the inverse InvertibleDict, it reuses the existing ones both for efficiency and so that operating on the inverse also modifies the original. However, if I inherited from dict I would necessarily be creating a new dict when I create the inverse, so instead I would probably have to create a "View" type in addition that implements the MutableMapping protocol, but that's already what InvertibleDict is.
@@mCoding I’m sorry, I should have been more elaborate: Yes, inheriting `dict` directly is often not a good solution. I meant my question more in this sense: 1) Does it make sense to inherit `dict` at all, are there any cases? 2) What problems lie in this direction? This would have made a great introduction to why the base classes exist in the first place and why inheriting them instead of the real thing is often the better choice.
#pycharm
1:45 I think the "I won't modify you" is wrong. Sequence, Set, and Mapping can be mutable. If you do "isinstance" with list and tuple against Sequence and MutableSequence, the only one that is False is tuple against MutableSequence.
Great observation and this is a subtle point. Type annotations are not checked by Python and so their meaning is highly dependent on who is using them. Different people can hold themselves and their team to different standards regarding how they are used and whichever standard you prefer to use is up to you. Read another way, it's often unspoken rules that dictate what they mean. Most people agree that b: B means b should be an instance of B, either literally in the sense of passing an isinstance check or structurally for ABCs and Protocols. So at the bare minimum d: Mapping means d should be a Mapping. As you point out, being a Mapping doesn't mean you aren't also a MutableMapping, so code that takes your Mapping, checks to see if it is a MutableMapping using isinstance, and then mutates it is technically allowed because you checked the type. However, there's a stricter level of interpretation for d: Mapping, that you promise to program only against the Mapping *interface*. If you hold yourself to the standard of programming against the Mapping interface, checking to see if your argument is a MutableMapping and then mutating it would certainly be unwelcome behavior :), even though it is not going to error at runtime. So a common unspoken rule here is that if you type hinted d : Mapping, you aren't going to call any mutating methods even if they exist, and that if you really wanted to mutate the mapping, you would have typed it as MutableMapping. Again, this is pure convention and many people do not abide by this convention, but that convention is what I was tacitly using here when I said "I won't modify you". It's not that I can't modify you, it's that if I intended to call any modifying methods I would have used MutableMapping.
@mCoding Oh, I see. That makes a lot of sense now. Thanks for the explanation.
What is he teaching in this video?
Why define a missing object variable instead of using None?
Great question! The reason is because the user may have intended to store the value None in the dictionary, e.g. {"a": None} or {None: "a"}. We need to be able to distinguish this case from the key/value not previously existing.
@@mCoding Aha, hadn't thought of that. But in that case, wouldn't it be more clear if we explicitly check if the key exists, instead of providing a default to get, and checking if we got the default?
Gimme #pycharm
something something #pycharm
The inv property is a little expensive though. It creates a new object every time it is invoked. If the dict has too many items, the dict comprehension used in __init__ can be a little expensive.
I don't have a better solution though.
#pycharm
I think that using a cached_property would work fine there
@@tylerfusco7495 maybe. But you would have to refresh the cache either time the forward or the backward dictionary is updated. The net effect depends on the ratio of reads to writes.
@@Mayur7Garg you would not need to refresh it, it would always be up to date, as there is no way to swap the underlying dicts (within the public interface). Everything is a reference, and the dicts are not copied for the inv property.
I would also argue that the overhead of creating a single python object, especially when slots are being used, is silly to worry about. Python will have larger overheads everywhere, so optimizing this part is not an effective solution.
If you really want to avoid the overhead of the extra class creation, i would add an extra field to the slots specifically for the cached inverse mapping. You would probably need to add extra complexity to __init__, but it would allow you to simply return the cached counterpart in the inv property. This would mean that for an InvertibleDict a, "a.inv.inv is a" returns True.
@@coarse_snad Yeah.. I was actually looking at the dict comprehension call but now I realised that won't be called during inverting. During the inversion, only references need to be updated since the backward mapping is not None.
3:08 "How restrictive this becomes for the caller of the function"
I have never heard of python ever enforcing type hinting. I could have "def foo(x: float): pass" and call with "foo(2.0)" or "foo([1, 2])" and get away with it. I've only ever heard of type hinting applying with LINTERS, not runtime.
Sure, someone reading the library code would go, "Oh, this function is expecting a float, not a list". Sure, but like pirates of caribbean, they're more like guidelines.
The types are indeed suggestions, for now, but they may not be in the future. Generally it's a good idea, and good practice, to listen to yourself if you suggest that something may be a specific type.
You're right that it is not the python runtime that complains, it is the static analysis tools like linters, IDEs, and type checkers. In the professional world, these static analysis tools are very much a part of everyday life and depending on the project, failing a type check in CI means failing the build. Additionally, regardless of whether this is an actual blocker, type hints are a form of communication of intent between programmers. If i type hint a function as taking an int, you should not pass a str to it. If i type hint a function as taking a dict, you should not pass a dict-like thing that doesnt inherit from dict. So whether this is enforced by your CI system or not, other programmers will be affected by restrictive type hints.
@@mCoding Just be more careful about your wording in the next video.
I was not saying anything about the merits of type hinting. The wording in your video made it seem like the langue enforces types: "How restrictive this becomes for the caller of the function". Maybe it was not clear as to how the caller is "restricted". Is in the human author? Is it the python run time engine? What is the "caller"? What is the "restriction"?
I live in a world where everyone seems to know what everyone else means, like an inside joke or whatever. Tear me apart for why I'm responsible for misunderstanding you, but at the end of the day I still lose out.
Just please, next time speak more clearly. Thank you.
Thích nhỉ.
early gang
0:43 - how did you get the dark theme python docs? Looks very usable :-)
Experimental chrome flag, although as it was kindly pointed out to me, the docs for Python 3.12 allow you to select your theme, and it looks much nicer.
Great joke on 15 series video
fn = InvertableDict({ 0: “Tokenizer”, 1 : “ #pycharm “})
## #pycharm #neovim #novs*code
Great video as always! #pycharm