Hi Arjan, thank you for your excellent video. Some comments: 1) As others already have pointed out, you really should not dismiss random testing for unit tests. With a fixed random seed, you can have determinism in regression and release tests. I would rather say, random testing is an essential testing technique for non-trivial software and multiple state-of-the-art variants such as "property based testing" exist. 2) There are many coverage variants such as path coverage, branch coverage, condition/decision coverage, etc. It would have been good to point out that you talk about line coverage which is a rather weak coverage metric. 3) Personally, I think line coverage is often overrated and if developers focus on having a high coverage too much, they tend to write poor tests that do not really verify the functionality, however simple increase the coverage number. It is good to find verification holes though.
Certain kinds of randomized testing have their place (even in non-medical device code). For instance, property-based testing, as exemplified by the Hypothesis library in python, lets you generate data according to a spec and then assert that the result of a piece of code obeys some properties. But the data spec and the properties are usually only discovered once you write a lot of manual unit tests, so what you've said here still applies. :)
Property based testing is an aweseome testing technique that I also wanted to point out. Hypothesis is a good "QuickCheck" clone for the Python language.
This is great, but I’d love to see how to completely cover functionality of an entire API! This includes API error handling, input validation, lower level database integration functionality and the business logic surrounding all this. Obviously testing the entire API call results in some seriously hefty tests and mocks so it’d be great to know how to split this out? Would you unit tests the lower level functionality, and test the api layer independently of this lower layer by mocking the lower layer call? Thanks
The last API I wrote was using GraphQL. It consists of a collection of functions and classes that perform basic database operations that make sense in the context of the application (e.g. if you delete a user from the database, you should also delete any other resources that the user created). You can test those functions separately, with a mock DB. The API then uses those lower-level functions, which are mocked again when you directly test the API functions. On the API level, you would mostly have to test that access to data by the requester is properly handled, and also verify that the data you send back only contains what you want it to. Ha, so that's basically what you wrote :). This is a nice idea for a video topic though, I will make a note of it and perhaps it is something I can cover in a future video.
Use the dependency injection instead of mocking api( or say db) functionality . A lot of mocking leads to test mock and not the actual code. Mocking should use wisely
@@alexign What's about fixtures? Your goal is to test your algorithm, not it's dependencies. Use the right tool and framework for the task. To test algorithms depending on databases or ORM use sqlalchemy for example or one of the many test libraries for several database variants to mock the external behavior.
Another quality video, thanks Arjan 😊 A small tip: instead of having to define your vehicle object for every unit test, you can define it once using the setUp() method under your test case class as follows: def setUp(self): # Camel case required self.v = VehicleInfo("BMW", False, 10000) You can then refer to self.v in all of your tests, and there's also a tearDown() method if needed
I think it might not be usable in every scenario. For example if the methods of vehicle class is modifying any data of instance I think you will need to create separate instances in each test case
Nice video! A quick note on your advice never to use stochastic tests. It's awesome if you can avoid it (obviously)! But since I have an electronics background myself I can tell you that most of the digital circuit verification is based on randomized tests. Once the number of possible code paths starts exploding you have to resort to stochastic code analysis. Same goes for very complex software. Certain critical applications like code that flies airplanes and runs on critical medical devices obviously needs to obey to a much higher standard.
Mutation testing can help to find issues with tests. The test framework will subtly modify your application during execution and check that your tests fail.
Use a fixed random seed and you can have determinism in regression/release tests for instance. Random testing is a powerful testing technique that can be combined with unit testing. No need to avoid it, I would rather say it is essential to test non-trival software.
Excellent summary of unit testing and TDD! Although, personally, I prefer to write one unit test at a time and pass it before moving on to the next one.
Your videos makes me want to sneak away and secretly code during the vacation with the family. If you have any tips on how to pull that off while still staying married that would be of great help :)
This might be an unpopular opinion, but i think the only really good thing with unit tests is that they are fast to run. Other than that, they suck because they generally just tests that the implementation is exactly as you programmed it, and it also tests that the bugs that are there still exists. What i think is more usuefull is to test you public api, which can be a web endpoint, a file or a set of methods, and then you mock away things like database and so on and try to get good coverage with that. Then you get tests that are stable. And the implentation can be whatever as long as it does not break the public api. Some people call these compontent tests. These are usually the only tests you need, and some times you also want to test really complicated methods. Thats all. Less tests, less shit to maintain. No need to change 100 tests when you refactor a class.
"Make your unit test 100% deterministic" Alrighty! But then generating thousands of random test cases *starting from a fixed RNG seed* should be preferable over just using one test value, right? Or would you argue that is more prone to introduce the same bug in your calculation in both the main code and the unit test, since the test value is (likely) calculated dynamically as well? Also, would you agree that testing cases that are "close to" edge cases is advisable? E.g. here, testing tax exemptions that are just above and just below (and equal to) the pre-exemption tax amount to be paid? Thanks for this video; I'm in data-analysis where there isn't a lot of versioning, but I'm still gonna add unit-testing to my Trello templates.
If you start from a fixed seed, then your tests are 100% deterministic ;). Testing close to edge cases is definitely a good idea as those are the areas where things potentially go wrong. Since releasing this video, I have 'evolved' my standpoint on random values in tests. In some cases random unit tests can actually be useful, for example to help generate test cases that the developer may not have thought of when writing the code. Some software testing tools use AI that automatically generates test cases and tries to actively find things that break in your code. This is a different approach that also includes randomness and is still useful.
@@ArjanCodes Exactly, agreed. I guess it depends on context; making an error/bug replicable while still being able to test a full range of 'random' cases would be quite valuable, especially in data-heavy applications. Thanks.
This is exactly what I also wanted to say. Random testing is a powerful testing technique which can be easily combined with unit testing and should not be dismissed. The determinism problem can be easily solved by using a fixed seed as stated above.
@@robertbrummayer4908 Yes, although I was wondering how you can make the test "independent". Suppose you have some function to calculate some value. For simplicity's sake, say y=2*x+3 (but imagine some complicated algorithm is involved). You could use a fixed case that you know to be correct. Say, if x=1 then y=5. You could test on such concrete known and verified cases. But suppose you generate some random value, say x=0.4. Against what y do you check then? You would have to be able to compute the function, dynamically, to arrive at y=3.8. But the whole purpose was to check whether you're calculating your function value correctly. That is circular, in a way. If your code to be tested has an error (maybe you misunderstood the algorithm), you cannot exclude that your test code has the same error.
No particular reason, but the programming language I'm most used to is TypeScript/JavaScript, so docstrings are not in my system (yet :) ). It is the recommended way of documenting functions in Python so I'll make a note of this and use docstrings in the future.
@@ArjanCodes I can recommend the Auto docstring generator VSCode extension. It adds all the boilerplate and uses the type hints and exceptions you added to your code and you just have to fill out the template
Hi Arjan, I'm having trouble running "coverage" in a setup where you have your package code and test code in different folders. The "vehicle_info" package is not found (nonetheless it works correctly in vscode). Any tips for using the "coverage" library in a different setup than this video (I believe the most common setup is to have test files and package files in different directories ?)
Hey Arjan. These are great videos you are producing, however, I missed the part where you described "test-driven development". Perhaps you could do a dedicated video on this topic.
What do you mean exactly? In the video, I’m planning to go through the TDD process (in particular the red-green-refactor part). If there’s anything in particular you’d like to see, let me know and I’ll see if I can add it.
In hardware design we have access to tools suites that can take test plans with requirements written in spreadsheets, and when writing unit tests you attach those requirements to tests. As the tests pass it checks it off on the test plan. Do you happen to know of something similar in Python?
That’s why it’s not enough to only look at coverage. Coverage only tells you which code has been run when you run your tests, not whether you’ve properly tested your code.
Thanks Reuben! This was in my browser, but you can preview html in VSCode using the following extension: marketplace.visualstudio.com/items?itemName=hdg.live-html-previewer. It does not support JavaScript, but for viewing generated html coverage reports, it should work just fine.
That is type hinting. Using the arrow after the function definition is telling readers of the code that "this function returns a boolean". In other languages, you have to declare the types of your variables and the types your function returns. Type hinting is an optional way in python to communicate to others what types are present in your code. See docs.python.org/3/library/typing.html.
It's seems odd that you chose to use unittest for these example, it seems much heavier weight than pytest to set up a test: you need to write it as a class, you have to inherit from the unit test classes etc etc. All this seems like a big disadvantage to using it in videos. Plus isn't it far less popular than pytest? What led you to choose unittest? edit: oh, nvm, I just realized you probably chose unittest because it's a Python builtin. It's funny that almost everywhere I go that references testing, pytest is listed first and unittest and nose later. It seems like, despite unittest being a builtin, pytest is far more common in the community.
Mutation testing can bring you a lot closer to 100%. But for non-trivial applications I doubt, that we'd ever reach true 100%. Of course 100% line/statement coverage, but not true coverage.
Here are also a few tips to help you write software faster: th-cam.com/video/MU20ah5s9ww/w-d-xo.html
_Testing can show the presence of errors, but not their absence._
-E. Dijkstra
Your videos are sooooo good, man! It pains me to such few views and subscribers
Glad you like them! The number of views/subscribers is low, but I also started this channel less than three months ago. TH-cam is a long game ;).
Hi Arjan, thank you for your excellent video. Some comments: 1) As others already have pointed out, you really should not dismiss random testing for unit tests. With a fixed random seed, you can have determinism in regression and release tests. I would rather say, random testing is an essential testing technique for non-trivial software and multiple state-of-the-art variants such as "property based testing" exist. 2) There are many coverage variants such as path coverage, branch coverage, condition/decision coverage, etc. It would have been good to point out that you talk about line coverage which is a rather weak coverage metric. 3) Personally, I think line coverage is often overrated and if developers focus on having a high coverage too much, they tend to write poor tests that do not really verify the functionality, however simple increase the coverage number. It is good to find verification holes though.
Certain kinds of randomized testing have their place (even in non-medical device code). For instance, property-based testing, as exemplified by the Hypothesis library in python, lets you generate data according to a spec and then assert that the result of a piece of code obeys some properties. But the data spec and the properties are usually only discovered once you write a lot of manual unit tests, so what you've said here still applies. :)
Property based testing is an aweseome testing technique that I also wanted to point out. Hypothesis is a good "QuickCheck" clone for the Python language.
This is great, but I’d love to see how to completely cover functionality of an entire API! This includes API error handling, input validation, lower level database integration functionality and the business logic surrounding all this. Obviously testing the entire API call results in some seriously hefty tests and mocks so it’d be great to know how to split this out?
Would you unit tests the lower level functionality, and test the api layer independently of this lower layer by mocking the lower layer call?
Thanks
The last API I wrote was using GraphQL. It consists of a collection of functions and classes that perform basic database operations that make sense in the context of the application (e.g. if you delete a user from the database, you should also delete any other resources that the user created). You can test those functions separately, with a mock DB. The API then uses those lower-level functions, which are mocked again when you directly test the API functions. On the API level, you would mostly have to test that access to data by the requester is properly handled, and also verify that the data you send back only contains what you want it to. Ha, so that's basically what you wrote :). This is a nice idea for a video topic though, I will make a note of it and perhaps it is something I can cover in a future video.
Use the dependency injection instead of mocking api( or say db) functionality . A lot of mocking leads to test mock and not the actual code. Mocking should use wisely
@@alexign What's about fixtures? Your goal is to test your algorithm, not it's dependencies. Use the right tool and framework for the task. To test algorithms depending on databases or ORM use sqlalchemy for example or one of the many test libraries for several database variants to mock the external behavior.
Just subscribed to your channel. You're helping me take my Python game to the next level! Learnt alot, keep on throwing out content like this please!
I really appreciate your videos Arjan. Great content. I predict a very fast growth for your channel. Keep it up. :)
So nice of you, Patrik! Happy you like the videos.
Great video, thank you for explaining how bugs can exist despite 100% coverage.
Thank you Daniel, happy you enjoyed the video.
Another quality video, thanks Arjan 😊 A small tip: instead of having to define your vehicle object for every unit test, you can define it once using the setUp() method under your test case class as follows:
def setUp(self): # Camel case required
self.v = VehicleInfo("BMW", False, 10000)
You can then refer to self.v in all of your tests, and there's also a tearDown() method if needed
Thank you, and great suggestion. I might do a series in the future fully focused on testing. This is a good topic to include there as well.
I think it might not be usable in every scenario. For example if the methods of vehicle class is modifying any data of instance I think you will need to create separate instances in each test case
Nice video! A quick note on your advice never to use stochastic tests. It's awesome if you can avoid it (obviously)! But since I have an electronics background myself I can tell you that most of the digital circuit verification is based on randomized tests. Once the number of possible code paths starts exploding you have to resort to stochastic code analysis. Same goes for very complex software. Certain critical applications like code that flies airplanes and runs on critical medical devices obviously needs to obey to a much higher standard.
Thank you! That’s a good point. In can definitely imagine that in those cases using randomized tests makes sense.
Mutation testing can help to find issues with tests. The test framework will subtly modify your application during execution and check that your tests fail.
Indeed. I've "evolved" my point of view on randomized tests in the mean time and I'll surely revisit this topic soon on the channel.
Awesome job with this tutorial series Arjan!
Thank you so much!
A great video as always, Arjan! Thank you.
Glad you liked it, @HampersUK!
I really like my randomized data. You've made a good point though, I gotta overthink that.
Use a fixed random seed and you can have determinism in regression/release tests for instance. Random testing is a powerful testing technique that can be combined with unit testing. No need to avoid it, I would rather say it is essential to test non-trival software.
Excellent summary of unit testing and TDD!
Although, personally, I prefer to write one unit test at a time and pass it before moving on to the next one.
Your videos makes me want to sneak away and secretly code during the vacation with the family. If you have any tips on how to pull that off while still staying married that would be of great help :)
Marry someone who also likes to code 😉.
To late for that 🤣
This might be an unpopular opinion, but i think the only really good thing with unit tests is that they are fast to run.
Other than that, they suck because they generally just tests that the implementation is exactly as you programmed it, and it also tests that the bugs that are there still exists.
What i think is more usuefull is to test you public api, which can be a web endpoint, a file or a set of methods, and then you mock away things like database and so on and try to get good coverage with that.
Then you get tests that are stable. And the implentation can be whatever as long as it does not break the public api.
Some people call these compontent tests.
These are usually the only tests you need, and some times you also want to test really complicated methods. Thats all.
Less tests, less shit to maintain. No need to change 100 tests when you refactor a class.
"Make your unit test 100% deterministic"
Alrighty!
But then generating thousands of random test cases *starting from a fixed RNG seed* should be preferable over just using one test value, right?
Or would you argue that is more prone to introduce the same bug in your calculation in both the main code and the unit test, since the test value is (likely) calculated dynamically as well?
Also, would you agree that testing cases that are "close to" edge cases is advisable? E.g. here, testing tax exemptions that are just above and just below (and equal to) the pre-exemption tax amount to be paid?
Thanks for this video; I'm in data-analysis where there isn't a lot of versioning, but I'm still gonna add unit-testing to my Trello templates.
If you start from a fixed seed, then your tests are 100% deterministic ;). Testing close to edge cases is definitely a good idea as those are the areas where things potentially go wrong.
Since releasing this video, I have 'evolved' my standpoint on random values in tests. In some cases random unit tests can actually be useful, for example to help generate test cases that the developer may not have thought of when writing the code. Some software testing tools use AI that automatically generates test cases and tries to actively find things that break in your code. This is a different approach that also includes randomness and is still useful.
@@ArjanCodes Exactly, agreed.
I guess it depends on context; making an error/bug replicable while still being able to test a full range of 'random' cases would be quite valuable, especially in data-heavy applications.
Thanks.
This is exactly what I also wanted to say. Random testing is a powerful testing technique which can be easily combined with unit testing and should not be dismissed. The determinism problem can be easily solved by using a fixed seed as stated above.
@@robertbrummayer4908 Yes, although I was wondering how you can make the test "independent".
Suppose you have some function to calculate some value. For simplicity's sake, say y=2*x+3 (but imagine some complicated algorithm is involved). You could use a fixed case that you know to be correct. Say, if x=1 then y=5. You could test on such concrete known and verified cases. But suppose you generate some random value, say x=0.4. Against what y do you check then? You would have to be able to compute the function, dynamically, to arrive at y=3.8. But the whole purpose was to check whether you're calculating your function value correctly. That is circular, in a way.
If your code to be tested has an error (maybe you misunderstood the algorithm), you cannot exclude that your test code has the same error.
Love the videos! Keep up the good work!
Why do you use comments above the function or method definition header and not docstrings?
No particular reason, but the programming language I'm most used to is TypeScript/JavaScript, so docstrings are not in my system (yet :) ). It is the recommended way of documenting functions in Python so I'll make a note of this and use docstrings in the future.
@@ArjanCodes I can recommend the Auto docstring generator VSCode extension. It adds all the boilerplate and uses the type hints and exceptions you added to your code and you just have to fill out the template
Awesome, thanks! Will install that right now.
Hi Arjan, I'm having trouble running "coverage" in a setup where you have your package code and test code in different folders. The "vehicle_info" package is not found (nonetheless it works correctly in vscode). Any tips for using the "coverage" library in a different setup than this video (I believe the most common setup is to have test files and package files in different directories ?)
Hey Arjan. These are great videos you are producing, however, I missed the part where you described "test-driven development". Perhaps you could do a dedicated video on this topic.
Hi Andrew! Thanks - glad you like them! I’m actually working on a TDD video at the moment, so it’s in the pipeline!
@@ArjanCodes Excellent news! Please tell me that its going to be real TDD from XP.
What do you mean exactly? In the video, I’m planning to go through the TDD process (in particular the red-green-refactor part). If there’s anything in particular you’d like to see, let me know and I’ll see if I can add it.
@@ArjanCodes That’s great! ‘Red-Green-Refactor’ is one of the parts I was hoping for.
In hardware design we have access to tools suites that can take test plans with requirements written in spreadsheets, and when writing unit tests you attach those requirements to tests. As the tests pass it checks it off on the test plan. Do you happen to know of something similar in Python?
great video! well explained
Thank you, glad you enjoyed it!
randomized unit test should have some use case I believe
Superbe vidéo thanks!
Thanks Jordan, glad you liked it!
Great video! Sugggestion: Can you turn down the like/subscribe sound effect? It was a bit loud
Thank you, and thanks for your feedback - I’ll watch the volume for the next videos!
Why do you prefer pytest? I’m also only familiar with it but this seems so much simpler. I wonder why more projects don’t use base unittests..
Thanks a lot, fellow SDE!
5:36 so even just importing from the vehicle_info module and having an empty test case actually "tests" all the class and function definitions?
That’s why it’s not enough to only look at coverage. Coverage only tells you which code has been run when you run your tests, not whether you’ve properly tested your code.
He is the real MVP
minimum viable product?
@@itachi2011100 most valuable professor
Great video! When you pull up the html.index, do you do that inside VSCode (and what extension is that?) or is that just in your browser?
Thanks Reuben! This was in my browser, but you can preview html in VSCode using the following extension: marketplace.visualstudio.com/items?itemName=hdg.live-html-previewer.
It does not support JavaScript, but for viewing generated html coverage reports, it should work just fine.
sorry if it's a dumb question but what does it mean on the method that "-> bool" because i never saw using arrows unless programming in R
That is type hinting. Using the arrow after the function definition is telling readers of the code that "this function returns a boolean". In other languages, you have to declare the types of your variables and the types your function returns. Type hinting is an optional way in python to communicate to others what types are present in your code. See docs.python.org/3/library/typing.html.
Exactly. I know it’s not exactly standard to do this in Python, but I think it helps clarify the intention of a function.
You Videos are awesome man !!!!
100% video coverage completed - 1 bug detected: the software development elders did not want to be mentioned.
Let’s hope they are in a forgiving mood.
It's seems odd that you chose to use unittest for these example, it seems much heavier weight than pytest to set up a test: you need to write it as a class, you have to inherit from the unit test classes etc etc. All this seems like a big disadvantage to using it in videos. Plus isn't it far less popular than pytest? What led you to choose unittest?
edit: oh, nvm, I just realized you probably chose unittest because it's a Python builtin. It's funny that almost everywhere I go that references testing, pytest is listed first and unittest and nose later. It seems like, despite unittest being a builtin, pytest is far more common in the community.
The software development elders are 100% true...
Mutation testing can bring you a lot closer to 100%. But for non-trivial applications I doubt, that we'd ever reach true 100%. Of course 100% line/statement coverage, but not true coverage.
You should introduce HYPOTHESIS for unit tests
Yes - it's on my list!
Nice content
Thanks so much Danilo, glad you liked it!
Great Video!
Thank you so much!
another great video, but that semicolon almost gave me a heart attack
Haha, yes, old habits die hard :).
Unittesting is only step 1 then there is component test and system test.
Wait, there actually are Software Development Elders? I thought it's a myth.
Great video Thank you
You are so welcome!
how about like 2% coverage...