Excellent vid - much clearer than the documentation. One feature worth emphasizing is the automatic testing code generation. That’s the big gun for creating a ton of test coverage.
I told you property-based testing is awesome :). Great video, Arjan! One approach I like a lot is to use a test oracle. When I have an optimized and therefore complicated implementation of an algorithm, I often also implement a simple version of the algorithm which is easy to understand, but not optimized and not efficient. I use Hypothesis to generate random input, run both implementations, and then assert that they return the same result.
What would be really cool would be a comparison video, first write a small project (idk, sudoku, battleship, whatever) and then write appropriate unit tests for it, and then do it again but using test-driven-development. To me it was hugely interesting how it influenced the way I design my code and how sensible the coupling between tests and code almost automatically becomes.
Another awesome feature of property-based testing and also Hypothesis is that the failure-inducing input is automatically simplified to give you a small and simple counter example. This is beneficial for debugging.
You bring up an interesting capability and much misunderstood feature of Python; Decorators. Perhaps this could be a topic for the future? Thanks for sharing!
Excellent video! I am using pytest quite often but didn't know about hypothesis. This would take my tests to the next level! Keep these videos coming, Arjan
Didn't know about Hypothesis but I was aware of property-based testing (i.e. verifying a property of the system more than asserting a specific response for a given input). In my experience I was using a combination of unitest and Faker: it seems this approach would be more structured and synthetic. Definitely worth to try in one of my next projects.
I love hypothesis! It's an awesome package and I think it should be used much more. However I recognise that it can be tricky to know when to use it if you are used to regular ol' unit tests.
Thank you so much for this video. I also had an excellent experience with hypothesis. You covered almost every case, probably except using map and flatmap to generate complex strategies. We found lots of very strange bugs thanks to this framework. Also property based testing is kinda mind blowing at first, but when it becomes a second nature, you even write regular tests much better :) Thank you so much for all of your videos, i enjoy every second of it. Best coding channel on youtube ❤
Great video! I’d love it if you could do a vid on unit testing, integration testing and data quality checking for a data engineering shop. I know it’s not exactly your wheel house, but curious what you’d come up with.
Your videos are simply awesome. Please provide us with more valuable information specially about hypothesis in depth (as u mentioned). Keep up the good work! ✌🏻
every time i read about something cool i can count on you to make a video about it within the next few months instead of trying it myself xD this will be one i have to add into the mix
I’m sold on the idea, but translating it to a real project and not the toy examples is difficult. I think I need a real OSS code base that does it well to demonstrate non-trivial uses in like half a dozen use cases. As a data engineer, I can see the value of treating the inbound data as a random variable in tests but the cost to learn hypothesis always seems a little too steep to implement in a real world scenario.
I'm on the fence about this, too. I very much like a data-driven approach to testing, as it allows you to collect real-world test cases and add failure cases over time without having to touch the actual test code. The library could be very useful for fuzzing, though, but I don't know whether *all* output of the given generator (e.g. "integers()") is actually tested. Would be quite irritating to have the test randomly fail if only a subset is drawn each time. OTOH the run time could be prohibitively long if the number of test cases becomes overwhelmingly large (think matrixes or lists). I guess the utility of the library depends heavily on the type of software you're working with.
@@totalermist yeah I looked into ScalaCheck which is the equivalent in that ecosystem which is based off the QuickCheck in Haskell. Apparently the strategies run a limited sample of values but hold the seed. So if there is a failure it spits out the failing value and the RNG seed to the failure is repeatable. I think I heard in the ScalaCheck equivalent, that when it finds a failing case it tries to reduce it in some way to the simplest failing case. Eg if -1234 fails it’ll try -1. But I also heard the strategies can be biased / weighted to try common failure cases. Like strings() will try an empty string more often than even odds. But yeah… a walk through of a real OSS code base with the author of said code base is the master class I am after.
The thing that blew my mind about QuickCheck is that when it finds a failing example it shrinks it down to the smallest possible failing example. There are methods to do this with custom strategies but I had a lot of trouble wrapping my brain around them. Can Hypothesis do this? If so, can you do a follow-up about how to shrink your own custom examples?
Great video, as usual, thanks! This topic was new to me and I would be very interested in a second video on this subjet! Maybe as a suggestion: apply this to a more "real life" example? Given a small simple project, what parts would you want to test with normal unit tests, what parts would you want to cover with property-based tests, where to do both? What is really the added value of those extra property-based tests over the classic unit tests (including parametrized tests)? Maybe there are typical things that are very commonly tested with property-based tests? Also, I guess looking at the code coverage is not really the point... is it? It's just some thoughts :) I really liked the video, I see that this tool is extremely powerful, but I'm not sure how to apply this to my projects...
Arjan.. I absolutely ❤️ your videos. You’ve rekindled my passion for Software development. I’ll definitely use it for QA activities.🔎 BTW can you create a detailed video on code documentation 📑and cover tools like Pdoc3 Sphinx etc. that would be phenomenal
Some pedantic comments ^^: test_negative_team_size should be called test_non_positive_team_size since 0 is typically considered neither positive nor negative. It is definitely not negative. I also propose to have a simple unit test for 0 since this is typically an interesting special case. If you just specify max_value to be 0 it is not guaranteed that 0 will be part of the generated random inputs.
Thanks for the feedback, Robert. Indeed, the naming you suggest is more precise. One way to add the 0 example is by using the @example decorator in Hypothesis and this will make sure Hypothesis includes it in the test data.
@@ArjanCodes Hi Arjan! The example decorator is even better than my proposal. In this way one does not need an additional unit test. Awesome! Best wishes from Austria :)
Hi, I Always follow your videos and find them really useful. I would really like a video about event driven architecture showing both event and command handling, I wasn't able to find any good one of that around the youtube ether..
Hi! Looking all your videos while I am in the train 😄 really great channel! I have a question: do you thing it’s a good practice to use something like ‘test()’ to generate values? I do not like this approach, since you don’t have the control of the inputs your are using to test your function. While writing I was thinking that maybe this type of testing can be part of the fuzzing testing that maybe can be separated from the unit one
How does hypothesis generate values. Since they can't try every possible values (too slow), does the library use a specific strategy to find corner cases ?
Sure. For example, if you have an API endpoint for searching documents with a skip and limit parameter, you could use Hypothesis to run tests with various values of skip and limit. Or test endpoints that expect dates with various date values, etc.
Let's imagine that we have a function than gets 2 arguments: string and integer and returns the boolean. For example, it returns True if the second argument (int) == len(first argument) Actually, I'd provide parametrized tests for the most common cases with the next structure: @parametrize("data, expected_result", [ (("Arjan", 5), True), (("Hello", 3), False) ] ) Could I do something similar with @composite for example? Or any usage with parametrized tests / fixtures / lazy_fixtures / ... Would be a great video, IMHO
Still not entirely sold on this.. the basic tests (encoding) pass also if I replace the functions with "return inp"... and even adding a type checking property test won't cut it either.. sure you don't care about what data to feed the function with, but you need to make sure that for a given input the function return the expected! Or am I missing something here?
hypothesis seems like a nice tool, will definitely try it in my next project. One thing I'm slightly confused about is that you suggest property-based testing as better than unit testing. I would call the kind of tests you wrote unit tests. Is unit testing really just tests of the form input -> expected output? Hard coding input/ouput examples sounds like a terrible way to write tests. Do people actually do that?
> unit testing really just tests of the form input -> expected output? I'm here to say, yes, that is what some, most, people think is unit testing. And even with property-based testing, you still have to write a good number of this variety of test. How else does the machine know what is "supposed" to happen, without re-writing all the code you are testing in the test itself?
There is fast-check: th-cam.com/video/a2J_FSkxWKo/w-d-xo.html Also jsverify: dev.to/glebec/property-testing-with-jsverify-part-i-3one Both don't use decorators, though. Might be a fun project to port the hypothesis library to JS or TS while keeping the decorator approach.
Video is cool and enriching. But I have an error when trying to run the hypothesis through the generation. "ValueError: source code string cannot contain null bytes".
You all know how to make the hypothesis generate the test file in UTF 8 instead of UTF 16 through pycharm. No matter how I change the file encoding in pycharm, it will always create the test file in UTF 16.
Does anyone know if hypothesis package is smart enough to avoid redundant tests OR does it always generate data randomly ? A redundant test has a implies relationship. For e.g: If a number is divisible by 4, then its obviously divisible by 2. Hence, i say the latter test is redundant.
That's only TDD in its strictest form. In practice, though, I've never seen it applied consistently. Property-based testing is more of an in-between of unit tests (e.g. what TDD is based on), BDD, and integration testing. If you only test single code units or functions, it's more towards the UT side, if you use it to test against specs (given-when-then), it's more BDD, etc.
Could somebody tell me how should test for an example a function to calculate sum of squares would look like: I got a function: def my_func(a, b): return a**2 + b**2 so now for tests @given(integers()) def test_my_func(a, b): assert a**2 + b**2 == my_func(a, b) so my question is what's the point of a test like it or how to test math inside python? Because for me testing math looks like copying that func to test and checking if they are equal. Ofc I am not asking normal unit test: @pytest.fixture(a, b, result, [(1, 2, 5), (2, 3, 13)]) def test_my_func(a, b, result): assert my_func(a, b) == result so If I think correctly hypothesis for testing math has no value right?
I agree that testing in this way doesn’t really help because a bug in the math is not found since it’s also in the testing code. What you could do instead is test for properties. For example, the sum of two squares is always a positive number or zero. So you could write a test that verifies that this property holds for a bunch of different a’s and b’s. I’m not a mathematician, but there are probably other properties of the sum of two squares that you could write tests for as well.
@Aflous, I think this was intended as a simple example to illustrate testing mathematics functions. In a real software application, you probably wouldn’t have a function like this.
@@ArjanCodes As a mathematician, another one is using the fact that (a - b)^2 >= 0, so a^2 - 2ab + b^2 >= 0, so a^2 + b^2 >= 2ab. But also the opposite, (a + b)^2 >= 0 so a^2 + b^2 >= -2ab, which helps if one of a or b is negative. Another idea is testing the sum of two integers is also an integer. Of course we could think about extending to floats, complex numbers, lists of a,b pairs, etc. and you would need to check for different properties.
![[#2024MAMA] G-DRAGON - 무제(Untitled, 2014)+POWER+HOME SWEET HOME+뱅뱅뱅+FANTASTIC BABY | Mnet 241123 방송](http://i.ytimg.com/vi/Ox29z5Nf1Uk/mqdefault.jpg)
💡 Get my FREE 7-step guide to help you consistently design great software: arjancodes.com/designguide.
Excellent vid - much clearer than the documentation. One feature worth emphasizing is the automatic testing code generation. That’s the big gun for creating a ton of test coverage.
I told you property-based testing is awesome :). Great video, Arjan! One approach I like a lot is to use a test oracle. When I have an optimized and therefore complicated implementation of an algorithm, I often also implement a simple version of the algorithm which is easy to understand, but not optimized and not efficient. I use Hypothesis to generate random input, run both implementations, and then assert that they return the same result.
Nice one!
What would be really cool would be a comparison video, first write a small project (idk, sudoku, battleship, whatever) and then write appropriate unit tests for it, and then do it again but using test-driven-development. To me it was hugely interesting how it influenced the way I design my code and how sensible the coupling between tests and code almost automatically becomes.
Another awesome feature of property-based testing and also Hypothesis is that the failure-inducing input is automatically simplified to give you a small and simple counter example. This is beneficial for debugging.
The amount of bugs hypothesis has uncovered on projects I've worked with makes learning it worthwhile
You bring up an interesting capability and much misunderstood feature of Python; Decorators.
Perhaps this could be a topic for the future?
Thanks for sharing!
Excellent video! I am using pytest quite often but didn't know about hypothesis. This would take my tests to the next level! Keep these videos coming, Arjan
Didn't know about Hypothesis but I was aware of property-based testing (i.e. verifying a property of the system more than asserting a specific response for a given input). In my experience I was using a combination of unitest and Faker: it seems this approach would be more structured and synthetic. Definitely worth to try in one of my next projects.
I love hypothesis! It's an awesome package and I think it should be used much more. However I recognise that it can be tricky to know when to use it if you are used to regular ol' unit tests.
Thank you so much for this video. I also had an excellent experience with hypothesis. You covered almost every case, probably except using map and flatmap to generate complex strategies. We found lots of very strange bugs thanks to this framework. Also property based testing is kinda mind blowing at first, but when it becomes a second nature, you even write regular tests much better :)
Thank you so much for all of your videos, i enjoy every second of it. Best coding channel on youtube ❤
Thank you for this! I was already kinda doing this, but hypothesis will make the tests clearer and easier to make.
You’re welcome! ☺️
I had never heard of property testing and found the subject very very interesting. I would really appreciate more videos on it!
This is way better than what I could gather from documentation, kudos!
Thanks so much Vladimir, glad it was helpful!
Great video! I’d love it if you could do a vid on unit testing, integration testing and data quality checking for a data engineering shop. I know it’s not exactly your wheel house, but curious what you’d come up with.
Love the videos Arjan keep them coming.. Can't wait every time!
On it 😉
The back lighting in your videos are 👌
Thanks!
Nice, Arjan. I like your clear way of explaining sometimes complicated concepts. You taught me Python development the right way!
Thanks! Very nice. Feels like hypothesis with the faker package could be an elegant way to create, test & stub data.
You’re welcome! Indeed, that’s an interesting combination to explore
Great video, congrats and keep posting!
Thanks so much Javier, glad you liked it!
Your videos are simply awesome. Please provide us with more valuable information specially about hypothesis in depth (as u mentioned).
Keep up the good work! ✌🏻
Excellent video would love to hear more about hypothesis!
On it, Hector ;).
Very good video! I saw hypothesis package one time in our company, but I didn't have time to dive into it. After that vid I'll back there for sure :D
every time i read about something cool i can count on you to make a video about it within the next few months instead of trying it myself xD
this will be one i have to add into the mix
At your service 😁
Thank you for this informative video. Could you make a video to showcase property-based testing focused in Django API testing?
I’m sold on the idea, but translating it to a real project and not the toy examples is difficult. I think I need a real OSS code base that does it well to demonstrate non-trivial uses in like half a dozen use cases. As a data engineer, I can see the value of treating the inbound data as a random variable in tests but the cost to learn hypothesis always seems a little too steep to implement in a real world scenario.
I'm on the fence about this, too. I very much like a data-driven approach to testing, as it allows you to collect real-world test cases and add failure cases over time without having to touch the actual test code. The library could be very useful for fuzzing, though, but I don't know whether *all* output of the given generator (e.g. "integers()") is actually tested.
Would be quite irritating to have the test randomly fail if only a subset is drawn each time. OTOH the run time could be prohibitively long if the number of test cases becomes overwhelmingly large (think matrixes or lists).
I guess the utility of the library depends heavily on the type of software you're working with.
@@totalermist yeah I looked into ScalaCheck which is the equivalent in that ecosystem which is based off the QuickCheck in Haskell. Apparently the strategies run a limited sample of values but hold the seed. So if there is a failure it spits out the failing value and the RNG seed to the failure is repeatable.
I think I heard in the ScalaCheck equivalent, that when it finds a failing case it tries to reduce it in some way to the simplest failing case. Eg if -1234 fails it’ll try -1. But I also heard the strategies can be biased / weighted to try common failure cases. Like strings() will try an empty string more often than even odds.
But yeah… a walk through of a real OSS code base with the author of said code base is the master class I am after.
The thing that blew my mind about QuickCheck is that when it finds a failing example it shrinks it down to the smallest possible failing example. There are methods to do this with custom strategies but I had a lot of trouble wrapping my brain around them. Can Hypothesis do this? If so, can you do a follow-up about how to shrink your own custom examples?
Great video, as usual, thanks!
This topic was new to me and I would be very interested in a second video on this subjet!
Maybe as a suggestion: apply this to a more "real life" example? Given a small simple project, what parts would you want to test with normal unit tests, what parts would you want to cover with property-based tests, where to do both? What is really the added value of those extra property-based tests over the classic unit tests (including parametrized tests)? Maybe there are typical things that are very commonly tested with property-based tests? Also, I guess looking at the code coverage is not really the point... is it?
It's just some thoughts :) I really liked the video, I see that this tool is extremely powerful, but I'm not sure how to apply this to my projects...
Arjan..
I absolutely ❤️ your videos. You’ve rekindled my passion for Software development. I’ll definitely use it for QA activities.🔎
BTW can you create a detailed video on code documentation 📑and cover tools like Pdoc3 Sphinx etc. that would be phenomenal
Some pedantic comments ^^: test_negative_team_size should be called test_non_positive_team_size since 0 is typically considered neither positive nor negative. It is definitely not negative. I also propose to have a simple unit test for 0 since this is typically an interesting special case. If you just specify max_value to be 0 it is not guaranteed that 0 will be part of the generated random inputs.
Thanks for the feedback, Robert. Indeed, the naming you suggest is more precise. One way to add the 0 example is by using the @example decorator in Hypothesis and this will make sure Hypothesis includes it in the test data.
@@ArjanCodes Hi Arjan! The example decorator is even better than my proposal. In this way one does not need an additional unit test. Awesome! Best wishes from Austria :)
Hi, I Always follow your videos and find them really useful. I would really like a video about event driven architecture showing both event and command handling, I wasn't able to find any good one of that around the youtube ether..
Haven't watched it yet but, yes hypothesis is amazing
I've seen this hypothesis in some project before. This is the first video approaching this package, btw
would love to see how this is used with dictionaries/pandas data!
Excellent!
Thank you!
Awesome! Thanks!
Happy you liked it!
Thanks a lot! Please cover `mutmut` next.
You’re welcome. And that’s also on the list. 😉
Hi! Looking all your videos while I am in the train 😄 really great channel!
I have a question: do you thing it’s a good practice to use something like ‘test()’ to generate values?
I do not like this approach, since you don’t have the control of the inputs your are using to test your function.
While writing I was thinking that maybe this type of testing can be part of the fuzzing testing that maybe can be separated from the unit one
I saw a great way to make hypothesis tests cleaner. When importing you do:
from hypothesis import strategies as some
And then
@given(some.text())
This is an awesome idea! I can't believe I haven't seen this before
Stealing this, thank you!
Arjan, did you have a video explaining how to log? I have a problem with logging my program with multiprocessing and filerotating handlers. 😢
How does hypothesis generate values.
Since they can't try every possible values (too slow), does the library use a specific strategy to find corner cases ?
What do you think about using hypothesis for testing an API? Would generating payload data using hypothesis help discover any bugs?
Why not?
There is schemathesis for APIs
Sure. For example, if you have an API endpoint for searching documents with a skip and limit parameter, you could use Hypothesis to run tests with various values of skip and limit. Or test endpoints that expect dates with various date values, etc.
Quite interesting
Thanks Kriss, happy you’re enjoying the content!
whats your favorite for software development ,windows ,linux or mac?
Let's imagine that we have a function than gets 2 arguments: string and integer and returns the boolean.
For example, it returns True if the second argument (int) == len(first argument)
Actually, I'd provide parametrized tests for the most common cases with the next structure:
@parametrize("data, expected_result",
[
(("Arjan", 5), True),
(("Hello", 3), False)
]
)
Could I do something similar with @composite for example?
Or any usage with parametrized tests / fixtures / lazy_fixtures / ...
Would be a great video, IMHO
I don't get the Python highlighting in VSCode... how the settings decorator is green but all the others are yellow.
ok. it does not verify that the code matches specification, but it tracks bugs by matching invariants of function and reciprocical functions?
Tangent: You've nailed the lighting, but the background sound is a little too loud and distracting, at least to my ears.
Still not entirely sold on this.. the basic tests (encoding) pass also if I replace the functions with "return inp"... and even adding a type checking property test won't cut it either.. sure you don't care about what data to feed the function with, but you need to make sure that for a given input the function return the expected! Or am I missing something here?
hypothesis seems like a nice tool, will definitely try it in my next project.
One thing I'm slightly confused about is that you suggest property-based testing as better than unit testing. I would call the kind of tests you wrote unit tests.
Is unit testing really just tests of the form input -> expected output? Hard coding input/ouput examples sounds like a terrible way to write tests. Do people actually do that?
> unit testing really just tests of the form input -> expected output?
I'm here to say, yes, that is what some, most, people think is unit testing. And even with property-based testing, you still have to write a good number of this variety of test. How else does the machine know what is "supposed" to happen, without re-writing all the code you are testing in the test itself?
i gave u a like bello 😂
Is there an equivalent for JS?
There is fast-check: th-cam.com/video/a2J_FSkxWKo/w-d-xo.html
Also jsverify: dev.to/glebec/property-testing-with-jsverify-part-i-3one
Both don't use decorators, though. Might be a fun project to port the hypothesis library to JS or TS while keeping the decorator approach.
Video is cool and enriching. But I have an error when trying to run the hypothesis through the generation. "ValueError: source code string cannot contain null bytes".
I found the problem. When use the terminal to generate the hypothesis, it will create a file with format UTF 16. Need to change to UTF 8 to work
You all know how to make the hypothesis generate the test file in UTF 8 instead of UTF 16 through pycharm. No matter how I change the file encoding in pycharm, it will always create the test file in UTF 16.
Does anyone know if hypothesis package is smart enough to avoid redundant tests OR does it always generate data randomly ? A redundant test has a implies relationship. For e.g: If a number is divisible by 4, then its obviously divisible by 2. Hence, i say the latter test is redundant.
I put my email in the code guide site box several times. Never came. Not in spam.
If you send me an email (support@arjancodes.com), I’ll make sure you get the guide.
Shouldn't you write tests before you code? Or is that only TDD?
That's only TDD in its strictest form. In practice, though, I've never seen it applied consistently.
Property-based testing is more of an in-between of unit tests (e.g. what TDD is based on), BDD, and integration testing.
If you only test single code units or functions, it's more towards the UT side, if you use it to test against specs (given-when-then), it's more BDD, etc.
Could somebody tell me how should test for an example a function to calculate sum of squares would look like:
I got a function:
def my_func(a, b):
return a**2 + b**2
so now for tests
@given(integers())
def test_my_func(a, b):
assert a**2 + b**2 == my_func(a, b)
so my question is what's the point of a test like it or how to test math inside python? Because for me testing math looks like copying that func to test and checking if they are equal.
Ofc I am not asking normal unit test:
@pytest.fixture(a, b, result, [(1, 2, 5), (2, 3, 13)])
def test_my_func(a, b, result):
assert my_func(a, b) == result
so If I think correctly hypothesis for testing math has no value right?
I agree that testing in this way doesn’t really help because a bug in the math is not found since it’s also in the testing code.
What you could do instead is test for properties. For example, the sum of two squares is always a positive number or zero. So you could write a test that verifies that this property holds for a bunch of different a’s and b’s. I’m not a mathematician, but there are probably other properties of the sum of two squares that you could write tests for as well.
@@ArjanCodes That totally makes sense :)
I don't get the point behind testing plain arithmetic functions, you need to trust the software at least for this basic task 😌
@Aflous, I think this was intended as a simple example to illustrate testing mathematics functions. In a real software application, you probably wouldn’t have a function like this.
@@ArjanCodes As a mathematician, another one is using the fact that (a - b)^2 >= 0, so a^2 - 2ab + b^2 >= 0, so a^2 + b^2 >= 2ab. But also the opposite, (a + b)^2 >= 0 so a^2 + b^2 >= -2ab, which helps if one of a or b is negative.
Another idea is testing the sum of two integers is also an integer.
Of course we could think about extending to floats, complex numbers, lists of a,b pairs, etc. and you would need to check for different properties.
Elon Musk explaining his Code after acquiring Twitter: 8:20
So this is pythons fuzzer.
It is _a_ Python fuzzer, yes.
👍 🐕
7:41 don't do that again pls