An approach I use is by going a level up to the function that is calling the one I'm about to write. In that calling function, the name and arguments should read in plain English, almost like a story. Ultimately, if you've got your face in a function you can see what it does, but if you are in the outer scope, a good signature means you don't need to dig deeper on that line to reason about what's going on. Great video, thank you!
Great video. At first I was wondering what could you possibly talk about 30 mins for just the function signature but I actually learned a lot. Thanks for putting this together
Calling it a "function header" is weird. The usual name you see for it is "function signature". That's what it's called in Python itself (see inspect.Signature).
Not feeling too comfortable with the “minus” in the function name. I prefer something like “calculate_total_after_discount” and then if the discount math changes the function name is still valid
There's another naming style sometimes used in the Python stdlib and ecosystem: adjectives describing the quality of the returned object, like reversed, sorted and functools.batched. They do not change the argument, so a name like "sort" would be confusing in this case, it's used for the corresponding method, which actually transforms the object. I don't know how this style is named though.
Arjan, great video as always, but I have something to add. A name 'calculate_totla_minus_discount' in my view is not so good. First, it describes too exact what it does, and second it looks like do many things: 'calc_total' and 'minus_discount'. I'd better name function 'calc_cart_total' or just 'calc_total' (may be 'calc_total_applying_discount'). There is no mention how it would be applied - 'minus' or 'plus'. For the user the name stays clear. Comparing dataclasses with TypedDict, I'd prefer dataclasses. The code with them at least visually is more clear: options.age_limit vs options['age_limit'].
...one more thing - about add_number_to_each_element. It does not clear from the name does it add in place or creating a new one. I'd prefer something like get_elements_increased_by
I mostly agree with you. The name should be concise, and if the behaviour is non-trivial, it should be documented in the docstring. I also think dataclasses are much better. TypedDicts are meant to be used to interface with older code that uses dictionaries for stuff like that, not for new code that can use dataclasses, Pydantic, or even NamedTuples. Regarding your comment about `add_number_to_each_element`, while I have problems with the name for being overly verbose, I think `get_element_increased_by` isn't that good either. Whether a function adds in place or creates a new one should be easy to describe: take the elements in as a Sequence or an Iterable, so you can't assign to the elements, and return a list. That shows the intent better. To be honest, my favourite name for this would be "increase_elements_by". The verb "get" is overused in function names.
thanks for posting. I'd rethink naming functions using their implementation detail. `calculate_total_minus_discount` is perhaps overly close to the implementation detail. I'd suggest something like `calculated_discounted_total_price`. that may read better in the caller's code too, as in ìf calculated_discounted_total_price(…):` I'd also argue about the benefit of verbs in functions names… after decades of using them! :) Writing in an FP language and everything is just data to me now :)
Also crucial: function arguments should always be annotated with the most general protocol possible, but the return type should always be as specific as possible type as possible.
Just quickly, the `def foo[T, U](...): ...` syntax is part of Python's newer generics syntax introduced fairly recently. It replaces the awkward use of type variables declared at the module level, where even if you wanted them just to be associated with a single function or class, they really weren't.
Great video as Always Arjan. Thanks for covering this topic. A real bugbear of mine is splitting function headers across multiple lines like @ 8:14 this is obviously auto-formated and it's a PEP8 guideline I know but I find it makes headers much more difficult to read (unless they are really long with lots of arguments - which they shouldn't be). I started using "autopep8.args": ["--ignore=E501"] in my settings to ignore long lines.
I also read that the function name specificity should be inversely proportional to its scope, e.g. a function collect_and_summarize_invoices might be used once or twice within the same class/module, but if a function is used all over the place, it should have a very short name, e.g. python's "open" function. thank you for another great video.
In general, I'd say you're better off writing straightforward signatures at first in the spirit of YAGNI. It's easy to spend too much time writing a perfectly generic function when you'll only ever use a single type with it. It's much better to start concrete and get more generic as you need to refactor. That being said, using Sequence/Iterable/Mapping doesn't hurt, as that's barely any effort, and you should return concrete types as much as possible. Finally, naming functions and parameters is an art. It's something I'm continuously thinking about. At the end of the day, you're better off documenting the behaviour in the docstring rather than trying to write the perfect name.
A tip I want to share which is slightly related is the “extract function” feature that a lot of IDEs have, which allows you to highlight a code block and press a hotkey to turn it into a function automatically. At least pycharm has this, and I guess you can find extensions for it for most of the popular editors. You can also do the inverse operation, meaning turning a function into inline code).
When handling default arguments, I often use the following line to set them at runtime: `timestamp = timestamp or time.time()` This works if the default value is `None`, because `None` is a "False" value and a valid timestamp is a `True` value. It also uses the fact that Python passes on the actual value of the first `True` element in an `or` operation, and does not evaluate the second value if the first one is already `True`. If the first value is `False`, it yields the second value regardless of whether it is `True` or `False`.
Something worth noting is that a better type annotation for generic numeric types is a union of numbers.Integral, numbers.Real, and decimal.Decimal (or numbers.Number if complex numbers are allowed as well).
Regarding variable naming, I use plural form for a collection as in "cars" unless the variable name hints to a collection as in "list_car", in which case I use the singular form.
You can also define your own types as variables, and the IDE will recognize them as doctypes (at least PyCharm does). Just define, for example: Real = int | float And use it as: def whatever_function(arg: Real) -> Real: ... It works for me in the latest version of PyCharm.
I can't write a function without type hints now, it's just automatic. They are worth using for the IDE hints alone IMO, in Neovim if I have set a function to take an int and accidently returned a string elsewhere, I know before even running any code. Saves a lot of time and frustration in our dynamic typing world.
Hi Arjan, thanks alot for the Video. What do you think about retuning a bool for functions that otherwise could return None? When using a bool as return one can control the main process by knowing If the function actually did work or would you argue that Program flow determines this anyway? Cheers
Minor nitpick: I think your analysis at 24:30 is not completely right. The reason for using a generic is to enforce that the type of the values in the returned list is the same as whichever type the user chooses to supply in the input Iterable.
(a) I do usually teach my guys to use "Iterator" when a (single) yield-statement is used in a function. The editor may detect Generator as the actual return type but it's not a good idea to be that specific. (b) I do also teach my guys to try to use "None" as the default as often as possible, and the actual non-nullable value can be set in a single line after the header - "value = value if value is not None else default". That's even more readable than a full if-block. In terms of string-values you would also want "value = value if value else default" anyway as usually an empty string is not an acceptable value.
When it comes to default values for options using a TypedDict you could define a privated options object and use dictionary merging, e.g. _default_options: Options = { 'foo': 0, 'bar': ['beep'], } def func(data: Data, options: Options) -> None: merged_options = _default_options | options
Tx. "Hardest thing"? Processing everyone's version of null, nul, Null, NULL, "null", \0, , None, Empty, "", 0, "0", "", [ ], { } and so on... esp found in modern, "low-code" data packets.
Great video. What about type hints of arguments which are types from other classes like a numpy array of Cosmology class from astropy for example. What would the best practice be for that? Just np.ndarray? Seems ugly.
Why? Type hints are type hints: they tell you what to expect and in an IDE, they enable accces to good auto completions. Using np.ndarray as type hints is super helpful when writing subsequent code in the function body because of type inference and intellisense autocomplete..but maybe I dont understand the term "ugly" in this context :)
As mentioned, there's numpy.typing to help with that. Unfortunately, there are many libraries that don't provide type hints, so sometimes you'll have to do manual casting (typing.cast) yourself. In some extreme cases, you'd have to provide typed wrappers around untyped libraries.
I disagree about the options object. It’s an approach that is very common in Java and C#, because those languages only know positional arguments, but in Python the configurable fields of the options object are more commonly passed as keyword arguments.
Maybe that's related to my programming history - I was tought pure C (K&R 2nd ed.) back in the late 1980ies - but how about abbreviating object names ? E.g. "calculate_total_price_including_discount" becomes something like "calc_ttl_prc_incl_dscnt" with arguments abbreviated similarly ? Is this an absolute no-go ? I hope not ... ;-)
I find list[int] typing unpythonic. It's nice to know what a function expects, but if you want many ints, use an array of int, where trying is both obvious and enforced. The point of a list is 2-fold: it's mutable, its elements are "any". The point of array.array(int, ) is that it's an ordered container of ints. I know it's not practical to implement, and no one uses the array from the standard library, so: j/s.
Even `calculate_total_minus_discount` is ambiguous. Is the discount subtracted per item? Is the discount a percentage of the total? If the discount is a percentage, should the user pass the percentage as an actual percentage (ie 25%) or a fractional proportion (ie 0.25)? Definitely the best function name would be `calculate _total_of_all_items_and_then_subtract_discount(item_prices: Iterable[int], total_discount_as_an_amount_of_money: int)`. If only there was a way to somehow leave a comment for a function that would document such particulars!
I exclusively use slotted dataclasses because of the performance benefits. Even if performance does not matter, either at all or in that area, I feel being consistent has more value than anything a dictionary can offer. Quick note on generics, the type parameter list in your examples was only added in 3.12 (if I remember correctly) and without those additions declaring type variables and manually handling variance is usually more mess and work than the value they provide.
Probably the most attractive aspect of Python used to be how simple it was to write it and to read the resulting code. One key part of that was duck typing - no need to specify what type of variable you were using, which also made it more flexible as eg the language would handle adding an int and a float. For some reason, people who like fully specifying types, and should probably have just stuck to those sorts of languages, have come along and fouled this up, now we are encouraged to write unreadable code using zillions of type hints. I was particularly amused, Arjan, with your section on 'making your function more generic' - achieved by adding even more type hinting ... you could just drop all the type hints and achieve that!
I am sure you have considered all the arguments pro and con type hinting already, so I am not going to change your mind. Let me just say that a significant portion of bugs in my project come from 3rd party modules not providing type hints or generic "Any" types. It takes way more time for a user of your module to crawl through documentation for debugging \ accepting all kinds of return types than referencing a typed interface.
I don't have half an hour to watch a video to see what is worth knowing or what is not. I am a speed reader and would like access to text versions of video.
I hate when someone requres to have everything with type hints. And I agree that they can make the code less readable. But It's hard to live without them
💡 Get my FREE 7-step guide to help you consistently design great software: arjancodes.com/designguide.
An approach I use is by going a level up to the function that is calling the one I'm about to write. In that calling function, the name and arguments should read in plain English, almost like a story. Ultimately, if you've got your face in a function you can see what it does, but if you are in the outer scope, a good signature means you don't need to dig deeper on that line to reason about what's going on.
Great video, thank you!
Great tip, thanks for sharing!
This is a great tip I’ve not heard before.
You can thank my old C++ days for that one! :)
"Make interfaces easy to use correctly and hard to use incorrectly."
- Scott Meyers, The Most Important Design Guideline
Great video. At first I was wondering what could you possibly talk about 30 mins for just the function signature but I actually learned a lot. Thanks for putting this together
Calling it a "function header" is weird. The usual name you see for it is "function signature". That's what it's called in Python itself (see inspect.Signature).
I guess that’s due to my upbringing in C! But yes, signature is correct in a Python setting.
@@ArjanCodes the first step to becoming a Pystro is forgetting all other languages.
Ah, there’s hope for me yet. I’m really good at forgetting things. 😁
Not feeling too comfortable with the “minus” in the function name. I prefer something like “calculate_total_after_discount” and then if the discount math changes the function name is still valid
@@DrDeuteronI thought the official term was "pythonista"
There's another naming style sometimes used in the Python stdlib and ecosystem: adjectives describing the quality of the returned object, like reversed, sorted and functools.batched. They do not change the argument, so a name like "sort" would be confusing in this case, it's used for the corresponding method, which actually transforms the object.
I don't know how this style is named though.
Arjan, great video as always, but I have something to add.
A name 'calculate_totla_minus_discount' in my view is not so good. First, it describes too exact what it does, and second it looks like do many things: 'calc_total' and 'minus_discount'. I'd better name function 'calc_cart_total' or just 'calc_total' (may be 'calc_total_applying_discount'). There is no mention how it would be applied - 'minus' or 'plus'. For the user the name stays clear.
Comparing dataclasses with TypedDict, I'd prefer dataclasses. The code with them at least visually is more clear: options.age_limit vs options['age_limit'].
...one more thing - about add_number_to_each_element. It does not clear from the name does it add in place or creating a new one. I'd prefer something like get_elements_increased_by
I mostly agree with you. The name should be concise, and if the behaviour is non-trivial, it should be documented in the docstring. I also think dataclasses are much better. TypedDicts are meant to be used to interface with older code that uses dictionaries for stuff like that, not for new code that can use dataclasses, Pydantic, or even NamedTuples.
Regarding your comment about `add_number_to_each_element`, while I have problems with the name for being overly verbose, I think `get_element_increased_by` isn't that good either. Whether a function adds in place or creates a new one should be easy to describe: take the elements in as a Sequence or an Iterable, so you can't assign to the elements, and return a list. That shows the intent better. To be honest, my favourite name for this would be "increase_elements_by". The verb "get" is overused in function names.
@@maleldil1 `increment_by(elements, increment)`?
well at this point, make a Cart class and put it in methods with a default _discounted=False class attribute.
thanks for posting. I'd rethink naming functions using their implementation detail. `calculate_total_minus_discount` is perhaps overly close to the implementation detail. I'd suggest something like `calculated_discounted_total_price`. that may read better in the caller's code too, as in ìf calculated_discounted_total_price(…):` I'd also argue about the benefit of verbs in functions names… after decades of using them! :) Writing in an FP language and everything is just data to me now :)
Also crucial: function arguments should always be annotated with the most general protocol possible, but the return type should always be as specific as possible type as possible.
Actually, the two hardest things in computer science is naming things, cache invalidation, and off-by-1 errors. 😉
LOL
It's not even a joke 👀
@@aflous which makes it extra funny
Haha, only serious
You could use NamedTuple for Options in that case it could be destructured almost like in TS
Can you point to more info on the syntax used where you have Numeric? The square brackets right after the function's name. Thanks!
Just quickly, the `def foo[T, U](...): ...` syntax is part of Python's newer generics syntax introduced fairly recently. It replaces the awkward use of type variables declared at the module level, where even if you wanted them just to be associated with a single function or class, they really weren't.
Great video Arjen, yes would love to hear your thoughts on function body design 👍
Great video as Always Arjan. Thanks for covering this topic. A real bugbear of mine is splitting function headers across multiple lines like @ 8:14 this is obviously auto-formated and it's a PEP8 guideline I know but I find it makes headers much more difficult to read (unless they are really long with lots of arguments - which they shouldn't be). I started using "autopep8.args": ["--ignore=E501"] in my settings to ignore long lines.
I also read that the function name specificity should be inversely proportional to its scope, e.g. a function collect_and_summarize_invoices might be used once or twice within the same class/module, but if a function is used all over the place, it should have a very short name, e.g. python's "open" function.
thank you for another great video.
In general, I'd say you're better off writing straightforward signatures at first in the spirit of YAGNI. It's easy to spend too much time writing a perfectly generic function when you'll only ever use a single type with it. It's much better to start concrete and get more generic as you need to refactor. That being said, using Sequence/Iterable/Mapping doesn't hurt, as that's barely any effort, and you should return concrete types as much as possible. Finally, naming functions and parameters is an art. It's something I'm continuously thinking about. At the end of the day, you're better off documenting the behaviour in the docstring rather than trying to write the perfect name.
Later equals never
A tip I want to share which is slightly related is the “extract function” feature that a lot of IDEs have, which allows you to highlight a code block and press a hotkey to turn it into a function automatically. At least pycharm has this, and I guess you can find extensions for it for most of the popular editors. You can also do the inverse operation, meaning turning a function into inline code).
If I’m not mistaken, starting with Python 3.7, the order of dictionaries is guaranteed.
When handling default arguments, I often use the following line to set them at runtime:
`timestamp = timestamp or time.time()`
This works if the default value is `None`, because `None` is a "False" value and a valid timestamp is a `True` value.
It also uses the fact that Python passes on the actual value of the first `True` element in an `or` operation, and does not evaluate the second value if the first one is already `True`. If the first value is `False`, it yields the second value regardless of whether it is `True` or `False`.
Loooved this video, so clear and helpful! Keep them coming!
Happy you liked it. Will do 😊
Something worth noting is that a better type annotation for generic numeric types is a union of numbers.Integral, numbers.Real, and decimal.Decimal (or numbers.Number if complex numbers are allowed as well).
Regarding variable naming, I use plural form for a collection as in "cars" unless the variable name hints to a collection as in "list_car", in which case I use the singular form.
the underscore notation is known snake case, just for curious, as well as camelCase has a name. Super good videos 👍🏻👍🏻
You can also define your own types as variables, and the IDE will recognize them as doctypes (at least PyCharm does). Just define, for example:
Real = int | float
And use it as:
def whatever_function(arg: Real) -> Real:
...
It works for me in the latest version of PyCharm.
Agree, type aliases are very handy for cases where the data type might change e.g. you want to change a string to a UUID
Man, what a beautiful video, I've learned a lot, thank you!
Happy to hear you enjoyed it!
Video was awesome can you make an other video on how declare value to a variable in depth .
I can't write a function without type hints now, it's just automatic. They are worth using for the IDE hints alone IMO, in Neovim if I have set a function to take an int and accidently returned a string elsewhere, I know before even running any code. Saves a lot of time and frustration in our dynamic typing world.
Hi Arjan, thanks alot for the Video. What do you think about retuning a bool for functions that otherwise could return None? When using a bool as return one can control the main process by knowing If the function actually did work or would you argue that Program flow determines this anyway? Cheers
Minor nitpick: I think your analysis at 24:30 is not completely right. The reason for using a generic is to enforce that the type of the values in the returned list is the same as whichever type the user chooses to supply in the input Iterable.
(a) I do usually teach my guys to use "Iterator" when a (single) yield-statement is used in a function. The editor may detect Generator as the actual return type but it's not a good idea to be that specific.
(b) I do also teach my guys to try to use "None" as the default as often as possible, and the actual non-nullable value can be set in a single line after the header - "value = value if value is not None else default". That's even more readable than a full if-block. In terms of string-values you would also want "value = value if value else default" anyway as usually an empty string is not an acceptable value.
Valuable insights. Thank you for posting.
Great one
I didn't know the typedDict, was struggling for awhile multiple optional field dataclass
Glad you enjoyed it!
Where I can find more info on this notation: def add_number_to_each_element_v3[Numeric: (int, float, Decimal)] ? Numeric is new to me.
What about NamedTuple for your Options instead of Dataclass? You can unpack a tuple.
Gosh you’re awesome arjan!
Hey Arjan, can you tell me how do you get this 'sparkles' indicator for the line that you're currently on?
It appears if you have the copilot extension installed. It allows you to access copilot if you click on it.
If you limit the function argument to few, will it trades off on dependency injection?
When it comes to default values for options using a TypedDict you could define a privated options object and use dictionary merging, e.g.
_default_options: Options = {
'foo': 0,
'bar': ['beep'],
}
def func(data: Data, options: Options) -> None:
merged_options = _default_options | options
Tx. "Hardest thing"? Processing everyone's version of null, nul, Null, NULL, "null", \0, , None, Empty, "", 0, "0", "", [ ], { } and so on... esp found in modern, "low-code" data packets.
thnx for the video, I've learned a lot from u!
You’re welcome! ☺️
What is your Keyboard Arjan?
Looks like NuPhy Air75.
Enjoyed ❤
That was a fire hose, but appreciated anyway!
Thank you!
If you send me an instance, I can access it attribute names and values in its dunder dict attribute. But that is some inappropriate intimacy.
Great video. What about type hints of arguments which are types from other classes like a numpy array of Cosmology class from astropy for example. What would the best practice be for that? Just np.ndarray? Seems ugly.
Why? Type hints are type hints: they tell you what to expect and in an IDE, they enable accces to good auto completions. Using np.ndarray as type hints is super helpful when writing subsequent code in the function body because of type inference and intellisense autocomplete..but maybe I dont understand the term "ugly" in this context :)
Numpy has a typing submodule to help a bit, though it's still in-progress (from numpy.typing import NDArray)
As mentioned, there's numpy.typing to help with that. Unfortunately, there are many libraries that don't provide type hints, so sometimes you'll have to do manual casting (typing.cast) yourself. In some extreme cases, you'd have to provide typed wrappers around untyped libraries.
@@lazerbro omg. what? my version doesn't have it tho. We got security lags.
@@mytelevisionisdead yeah I agree. It just looks ugly to me. I still use it.
Thanks!
Thank you so much!
What the function!
Great insight on important task !!
Thank you, glad you enjoyed it!
I disagree about the options object. It’s an approach that is very common in Java and C#, because those languages only know positional arguments, but in Python the configurable fields of the options object are more commonly passed as keyword arguments.
Maybe that's related to my programming history - I was tought pure C (K&R 2nd ed.) back in the late 1980ies - but how about abbreviating object names ? E.g. "calculate_total_price_including_discount" becomes something like "calc_ttl_prc_incl_dscnt" with arguments abbreviated similarly ? Is this an absolute no-go ? I hope not ... ;-)
You like the 'typing' module but it seems that the ' typing module is getting deprecated in Python' :
th-cam.com/video/cv1F_c66utw/w-d-xo.html
I find list[int] typing unpythonic. It's nice to know what a function expects, but if you want many ints, use an array of int, where trying is both obvious and enforced. The point of a list is 2-fold: it's mutable, its elements are "any". The point of array.array(int, ) is that it's an ordered container of ints. I know it's not practical to implement, and no one uses the array from the standard library, so: j/s.
Even `calculate_total_minus_discount` is ambiguous. Is the discount subtracted per item? Is the discount a percentage of the total? If the discount is a percentage, should the user pass the percentage as an actual percentage (ie 25%) or a fractional proportion (ie 0.25)? Definitely the best function name would be `calculate _total_of_all_items_and_then_subtract_discount(item_prices: Iterable[int], total_discount_as_an_amount_of_money: int)`. If only there was a way to somehow leave a comment for a function that would document such particulars!
I exclusively use slotted dataclasses because of the performance benefits. Even if performance does not matter, either at all or in that area, I feel being consistent has more value than anything a dictionary can offer.
Quick note on generics, the type parameter list in your examples was only added in 3.12 (if I remember correctly) and without those additions declaring type variables and manually handling variance is usually more mess and work than the value they provide.
=D
Probably the most attractive aspect of Python used to be how simple it was to write it and to read the resulting code. One key part of that was duck typing - no need to specify what type of variable you were using, which also made it more flexible as eg the language would handle adding an int and a float.
For some reason, people who like fully specifying types, and should probably have just stuck to those sorts of languages, have come along and fouled this up, now we are encouraged to write unreadable code using zillions of type hints.
I was particularly amused, Arjan, with your section on 'making your function more generic' - achieved by adding even more type hinting ... you could just drop all the type hints and achieve that!
I am sure you have considered all the arguments pro and con type hinting already, so I am not going to change your mind. Let me just say that a significant portion of bugs in my project come from 3rd party modules not providing type hints or generic "Any" types. It takes way more time for a user of your module to crawl through documentation for debugging \ accepting all kinds of return types than referencing a typed interface.
This programming nitpicking is getting ridiculous
Dont why so much work. Make it (/, **kw)
And let thr user decide what parameters he wants. Im too old for that
I don't have half an hour to watch a video to see what is worth knowing or what is not. I am a speed reader and would like access to text versions of video.
I'll be honest. I absolutely hate type hinting. It makes an unnecessary mess and makes it harder to read while not bringing any tangible benefit.
100% 👍
I hate when someone requres to have everything with type hints. And I agree that they can make the code less readable. But It's hard to live without them
At 12:44 instead the if statement I personally like timestamp = timestamp or time.time() better. Its looks cleaner..
But as always great video!
cyclomatic_complexity -= 1
ftw.
24:32 and what should we do if we want to combine different types in an iterable?
Very good tips, thanks for sharing.