The Lies Of 100% Code Coverage | Prime Reacts
ฝัง
- เผยแพร่เมื่อ 1 ต.ค. 2024
- Recorded live on twitch, GET IN
/ theprimeagen
Become a backend engineer. Its my favorite site
boot.dev/?prom...
This is also the best way to support me is to support yourself becoming a better backend engineer.
Article link: www.code4it.de...
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Kinesis Advantage 360: bit.ly/Prime-K...
Hey I am sponsored by Turso, an edge database. I think they are pretty neet. Give them a try for free and if you want you can get a decent amount off (the free tier is the best (better than planetscale or any other))
turso.tech/dee...
![ลองสั่งของจากแอพ temu ถูกจริงหรอ?? [โกงไหมครับ ep.88 ] | DOM](http://i.ytimg.com/vi/RTcZSQTiDZw/mqdefault.jpg)
Code coverage: Goodhart's Law all over again. "When a measure becomes a target, it ceases to be a good measure"
The only value of code coverage is during development: if your code coverage has dropped (significantly), you have _definitely_ missed _something._
Tests are not for detecting bugs. Tests are for checking if a function does Z when given XY.
But in most of the times it cant detect bad programming, race conditions. Also tests are written from the same person which has maybe wrote the bug in first place.
Write tests, but don't expect them to detect bugs.
Right. Tests exist to verify that for a given scenario, the code executes as expected. It doesn't even mean that it performed as expected. Just the result is as expected. It could entirely be by coincidence that the correct result was provided for that input.
I do like property testing as it helps with providing a greater range of inputs for verifying outputs.
This is a good argument for introducing the tests when refactoring the code. When you just wrote a function, obviously it does what it's supposed to do because you made sure it does. But when changing things around inside, it's easy to accidentally break something and tests help you with that.
The answer is obviously > 200% coverage.
that's genius... let's test unexisting features, so that they are already tested when the client asks for them !
Automating testing of tests… tech lead level thinking right here.
test driven development but we promise twice as many features as desired!
I'll tell you what, I'm covering something else to a 100% if you catch my meaning
pepe> gottem
deez?
That can be implemented in two different ways, which one is it?
100% agree with the "focus on integration tests over unit tests" opinion in the article. Unit tests often feel like testing for the sake of testing whereas my clunky and bulky robot_framework test suite finds stuff to fix or improve quite often
It’s crazy how integration tests were not even mentioned. Unit tests are not design to test end to end functionality of your application. It suppose to test individual “units” of it. In reasonable well designed applications 100% coverage would not be even possible because guess what? You have an entry point where you create all your dependencies and that hard and inefficient to mock. That’s when integration tests come in the picture. You should have a set of test that will call real dependencies. For example call the real weather api and verify that the output you get its correct. Units tests by design are not meant to test this type of end to end scenarios. If you find yourself in a situation like this, something in the architecture of your app/ service can be improved.
0:00 starts yapping
21:41 ends yapping
👏👏👏 to both our contestants on "when does it start and stop?"
Probably useful for those who hate Prime. Idk
@@XDarkGreyX We love Prime, what do you mean?
Gottem 😂
I'd like to see a mock testing framework _mock_ old school human testers; pun intended. Walk to a pub. Ask for 1 beer, -1 beer, 0 beer, 2^32 beer, 0.5 a beer. Come in through the window, exit through the back door. Breakdance inside and then defenestrate yourself. Introduce yourself as null, empty string, a null terminated string, CRLF, etc.
I believe the Haskell library QuickCheck does exactly that. I want a similar library for my languages of choice, which is not Haskell.
And it will still break as soon as somebody asks "is there a restroom?"
@@veryevilhumna oh of course! That's software. It's held together by tears, sweat, luck, and continued sacrifices to the silicon gods. It's amazing that we have software at all, much less software that is mostly bug free.
@@veryevilhumna preferably not but I can imagine someone asking for a MRI machine even if you handle restrooms. You can never really win
@@yramagicman675 haskell mentioned
The best code coverage is 100% if it’s achieved via tree shaking removal of all code that isn’t covered in an integration test.
Most unit tests are actually useless. I only add unit tests for specific functions that contain business logic. I have found to be better to write down every single possible scenario to go wrong and try to break it
This kind of policy just makes the test suite run all the lines of code but it doesn't make you test anything.
Gottem 🎉
It tests if you made a type in garbage language such as python.
I swear to god as someone who has been doing this professionally for less than 6 years. So many of the titles of your videos make me chuckle internally( and sometimes externally ). Because it's all things that, the realization of how bullshit some of these are, is still so fresh in my mind.
LESS than 6 years, wow. I've been doing this for less than 100 years.
@@flflflflflfl So weak, I have been doing this for less than infinity.
This is the first time I have seen someone using "less than" for describing their experience 😂
@@flflflflflfl I have been talking like this for less than 72 months ;)
What I've seen in relatively high integrity code is that code coverage hovers around 70-80%, mainly for one reason: defensive programming. It's good to be careful and future-proof by having plenty of basic checks (null pointer, bounds checking, integer overflow) and you can easily end up with like 20% of the lines of code being "dead code" because they check conditions within helper functions that are not called anywhere in a way that could trigger those errors. But that could change in the future, making those checks useful in the long run. Trying to achieve test coverage of all these checks is so tedious and high-maintenance that it discourages you from writing them in the first place.
Why are you checking if something is Null, you should guarantee it by the caller. Your code should segfault when used incorrectly.
I like to follow the pattern of appending "_unsafe" to all the functions that can explode when used incorrectly, then you can safely call unsafe functions from unsafe functions and wrap them all into a function that does safety checks if necessary.
@@lollertoaster I agree that the "_unsafe" pattern can be helpful. As far as checking for null (or other similar pedantic conditions on parameters), that is sort of the point of defensive programming. You might be able to trust today's caller (probably yourself), but you should not be so trusting of the future callers. How many security vulnerabilities have we seen over the years that started off as a little helper function that didn't bounds-check because it was only used in one "safe" place, but later was carelessly reused in other places? And sure, "_unsafe" helps with making that less of an easy mistake.
@@lollertoaster Your both points are contradicting each other. You state code should segfault if used incorrectly and at the same time, you propose to append "_unsafe" to functions that will segfault if used incorrectly which means any non-"_unsafe" functions are expected to be resistant towards misusage. What is it you are proposing, A or B?
If "_unsafe" functions are only used internally and are never going to be public, then it also makes no point to name them "_unsafe".
I agree btw that null checks should be avoided but if you're developing a library that expects the user to follow a non-implicit initialization of objects to be used, I'd rather say that you should avoid this kind of requirement and make sure that in any way your library can be used, that there isn't any way to make it segfault/throw unhandled exceptions in the first place.
100% value delivered "to the people" > 100% not happy path code coverage > 100% happy path code coverage
9:00 chatter seemingly doesn't realise: by US law, 100% code coverage, even if it's 100% by useful tests, *is not enough*. avionics have to be mathematically proven correct at every level from the hardware to compiler to actual code itself, even taking a whole range of failures into account.
FPV babyyy
Goodhart's law: "When a measure becomes a target, it ceases to be a good measure"
Easy to create shitty tests just because you have a shitty target.
High code coverage does not catch all bugs. One of the most common bugs is that the software doesn't do what it is supposed to do. You might get 100% code coverage because you are just missing the code that you are supposed to have and you aren't testing it either. Maybe a function is supposed to call an external service as one of the steps, doesn't care about the result. And if the function doesn't do it and the unit test doesn't check for it either, everything seems OK.
Better to just write tests for the more important stuff. Don't waste time on covering every edge case. Testing it manually if often enough. Tradeoff of spending time vs shipping faster.
300% coverage, bro. You got to pump those numbers up !
Just run your tests three or more times depending on how much coverage you get per run. It makes sense, cause the shitty tests are flaky anyway.
19:20 "The best way to have a well running project is to have tests easier to write and run to test your project than running the project itself" - this is the core takeaway from the video and it is absolutely true.
Identifying and managing the path of least resistance is the core of driving human behaviour in general.
I've recently encountered a bug when I was splitting a string to get the filename and the extension. It turned out that on production the server returned something completely unexpected and my function was not working correctly.
The function was 100% covered but still still failed so... where's the issue? well.. the implementation and the unit tests were poorly written, I made assumptions, I mocked values and I did not have enough context and knowledge.
100% Coverage, if done right, is ALWAYS a good thing. Don't believe anything else.
Hot take: Unit tests are, by definition, lazy tests. The only reason people do unit tests instead of benchmarks and end2end tests is that unit tests are quick and integration+load tests take some forethought. However, at the end of the day, what really matters is that your application is meeting functional requirements in a timely manner, not that you've covered every single (or even any) of your catch blocks. Moreover, as you get better at programming to meet functional tests, it will become easier and easier for you to spend your mental energy on thinking of possible failure modes FOR THE APPLICATION and writing the integration tests to cover those modes.
The only exception to this is when you are writing a piece of code that is too hard for you to validate in your head.
I agree. I think unit tests are actually useful if average developers have to work with a big system and regularily need to touch/maintain new code areas they dont know yet. but imo thats then a problem of the organization structure.
The problem with unit tests and code coverage is that it should only be part of the testing solution, not the entire testing solution as many companies want. As developers we are terrible at testing code because we only want to see the code work. A good tester likes it when he can cause your app to crash the machine. He tests for how to break the code in addition to "happy path" testing. The problem is that unit tests generally only test for happy path and then rest of the testing is passed on to the users. Most users will only test happy path and everything will seem hunky dory until someone finds that one evasive bug that brings down the entire system or worst yet spills all of your sensitive data out to the dark web.
Code coverage does not ensure the test actually verifies the results of the code being covered.
Mutation Testing mitigates that significantly
100 % code coverage is primarily a design tool. If you have achieved 100 % code coverage then you have run all lines of your program and you have proven that all of your code is testable and reachable in tests. That means more maintainable code. While codebases that does not have 100 % code coverage might be as good or better in this regard, the difference is that yours definitely has that property. Reminds me of Rust vs C++. You can write safe code in both, but the difference is that in Rust it definitely is safe.
"If a piece of code isn’t covered, it will likely contain bugs." is very true. What are you talking about?
Maybe the wording should be "more likely, but still
I would ask, why aren't testers writing tests? And i'm talking more SDET, actual developers in test... There is a completely different mind set between devs and dedicated testers....
I just started my first job at an extremely high security sector and the project i've been assigned needs to have as high as possible coverage or hundreds of lives can be lost due to one bug. So it depends a lot on what you're actually doing.
Tech lead/CTO: we need 100% code coverage.
Me: i don't think that means what you think it means.
Ideal: we will write tests that cover every use case and scenario and have tests that removes the QA team.
Reality: Programmers write shitty tests that noop to run every line and assert(true) at the end. Giving zero benefit. Or you end up with unmaintainable amount of tests that are commented out so that the test runner passes.
I have seen all of this, and you clearly have too. It’s a crying shame. Management and engineering should not have an adversarial relationship, I believe that would alleviate much of this problem.
Insightful video. Mutation Testing can cover the largest coverage risk you mentioned.
Writing good tests is what defines you as a developer. With 2 decades of experience, I can tell you I spend 80% of my time devising and writing tests. I reject the "unit test" because in the vast majority of scenarios they are pointless. I reject 100% code coverage because it will lead to people writing tests for getters and setters, and that is indeed the way to make youngsters reject testing.
It's also why I reject 10x developers
Btw, fully with you on "you've now tested your mock". WTF is that? That tests their Controller. So guess what? That dude/dudette now has to write two more tests: to test the Service and then to test the Real repository. Because , guess what: they're going to want to store stuff in a database and then find out that their Real query causes a deadlock... which they never saw because they were using a Mock.
If you code system with a database, especially if you use transaction, you do not mock the database layer away.
excellent points
Propose a better solution for test metrics instead of saying the current best practice of 100% coverage is bad. Bad compared to what? There's a reason why it's the current standard. Complain into a pillow so your poison is contained.
"Complain into a pillow so your poison is contained." 🤣🤣🤣🤣
There used to be a C# tool called Pex that analyzed the edge condition of your code and generated inputs to hit those edges.
For some reason it went the way of the dodo, probably was bundled into some "premium" feature that no one even knows exists.
But yes, its not "coverage" that is important. Its how many possible edge conditions you have tested. Here is the problem: The number of edge conditions is combinatorial with the complexity of your code. So at some point you need to turn to integration tests anyways. Honestly, with experience I grow more fond of only doing unit tests for algorithmic stuff and the rest just E2E balls to the wall testing the actual deployed thing.
I do 100% code coverage just to spite my coworkers for forcing me to make unit tests. Then they're stuck unable to commit anything that would lower code coverage and they hate tests too.
🤣🤣
21:35:
e2e tests is all integrations together.
Integration tests are tests done with services integrated.
If your system is very shallow (low amount of A needs B needs C needs D, etc... for a full execution) there's no distinction between e2e tests and integration tests.
Honestly, I've been saved on a few occasions by the "doesn't add any value assert(1).equal(1)" test. It's far from perfect, but if pressed by time, it at least gives a test checking the code is not dumbly crashing in the covered section.
7:00 Man, I would love to test my React apps without mocking. I would love to find guides telling me how to do, when the Google is full to the brim of "how to mock library X for React-testing-library". I would love for someone to finally give me a proper guide to how to properly test front-end apps that does not feel like a bunch of general rules and hand waving.
That's simple just call the backend and assert that things are working.
@@demmidemmi BRB, need to set up docker.
MSW
@@elgalas Manageable, but:
a) A bit of a pain in the ass to set up (on the data side of things, using Faker and such. Very easy to make your life harder when using without a deeper thought),
b) You are still mocking, you are just replacing proper / local server with proper / local database with entirely fake one,
c) At the end of the day you are faking backend data. Which is fine, but it is only a tip of the iceberg and not the most important one.
@@Salantor or... You could write dev time interceptors that save some of the data to disk.
Nevertheless, http calls to pull data, are manageable. HTTP transactions that depend on actual business logic in a backend, or mocking lack of browser APIs are real pain.
No amount of code coverage can compensate for bad design.
The best tests are those that find bugs in the code you just wrote, prevent bugs from appearing in the future, and communicate intent about the code under test.
Tests that do that will naturally have a good coverage metric, but code which has a good coverage metric may never find or prevent a bug.
my point of view (in my previous job the target was 100% code coverage):
- it only works if you know how to write testable code. If you don't, this target will be really, really time consuming
- if you try to achieve 100% code coverage, I've seen developers write tests for exception classes by creating the object and then assert the object is an instance of this class (i'm not kidding, they actually did this there)
- deleting untested code increase code coverage, deleting tested code decrease code coverage, so you do not remove tested code. Some even make their CI fail if the coverage decreases too much
- some people even go far to not measure code coverage over integration tests, so you have to write unit tests to cover the code, but to me an integration test also covers code and might make it even easier to find why some code exists.
- I also see that people write tests with names like method_doesSomething_works_as_intended, or testDoesSomething when they see the function doesSomething is not covered.
- I also see people use reflection or make private methods public (with a docblock marking it as @private or @internal) on it to test a private method (you should have moved it to it's own class maybe) to get all the testcases of private methods.
If you don't have one of these pitfalls and know how to write proper tests, you are fine with a 85-95% code coverage
Biggest problem for me with code coverage is that it gives a false sense of test quality and the more you force people to abide by it the worse it gets.
You want as good tests as you can get for the time you a lot to creating/running and that should be the end goal not coverage.
I'm sad that there was no mention of TDD in this.
Damn mockists, you write tests like this?
err := json.Marshal(anything)
you write if err == nil. then how would you test the err condition. So far I only was able to do it by passing a channel in anything. So if I have Marshal at 100 places then 100 test case with chan just to get to err case for 100% coverage. And I am still not sure how stupid a dev has to be to pass a channel to json.Marshal
I believe tests should be testing that a piece of functionality is working as intended and I'd go as far as saying that piece of functionality should work as intended in as nearly real-life situation as possible. Otherwise, we're just testing a mock for the sake of testing something without putting thought into it.
I find that x% code coverage demands usually just lead to bullshit unit tests (assert 1 == 1), unit tests that amount to testing that built-ins or libraries do what they say they do, or both.
And, yeah, mock is a big code smell. Sometimes it's necessary, but every time you do it, you should feel a little sad and hollow inside.
🤣🤣🤣🤣
Have I been banned or something. Or is TH-cam bugging? I think I've only commented a few times on this channel.
The thing is, yes, 100% as a goal is stupid, it is not stupid because of the coverage but because of the goal.
Thriving for anything but 100% makes zero sense in all worlds. So if code coverage is a metric demanded, making it 100% is the only valid goal. That, or ignoring code coverage entirely
@ThePrimeTime, I disagree with so many of your statements. If you cover the code with tests, then it is highly likely there are bugs IFF the tests were developed with care. I have spent many many many hours of my life creating and testing systems. Creating custom test sw, including interfaces. Everything you stated about short comings with tests just showed the time, and care are given or allowed. Also, if 100% code coverage is only by lines I do not agree. You should cover 100% of branches.
Reading this articles gives me vietnam flashback. This article is very clearly written by a PM or a exec who has no business writing code. They are treating code like a property, a inventory, without understanding how it works. But unfortunately, that's how the world works, across all industry, everyone all judge things they don't understand or have no interest in at a weird handwaivy level. Medicine comes into mind, FDA approval comes into mind, I can list more.
Unit-test means mock everything, **and if a single thing is not mocked then it's an Integeration-test**, finally, e2e-test's logic is a higher level of integeration-test where said test's logic clicks buttons and reads resulting dialog.
P.S. We love Integeration and e2e, but hate Mocking non-sense, which expects dependency to do exactly what it's Mock does.
I disagree with article statement about Code Coverage information.
The article states Code Coverage helps verify "the part of your code exercise by the tests".
I see this as a misunderstanding of Code Coverage information.
The information, in my understanding, is:
"What part of the codebase I have a risk of NOT finding an error until manual test or deploy in production?"
Therefore, the information is a list of methods, never executed by the test suite, and where they are defined.
Everything else is noise.
If you dont know what to test for with 100% coverage, then there are definitely some bugs you missed.
We do C# where I'm at (I know...). It's pretty common to have a WebAPI controller that has a dependency on some service and that service class has a dependency to some repository interface. When we execute a function on the service it will basically say "well first I have to use the repository to get whatever, and then I do this thing to it." How would you remove that dependency? I could move it to the WebAPI controller and pass the data to the service function but the other thing we are trying to do is keep WebAPI controllers as simple as possible and be just an IO device into the business rules.
Is there any article or video that someone could point me to that explains how to do a refactoring like that?
I think the author is pre-empting the points about good code coverage with the sentence prior.
If your tests are "valuable", then 'all these points are true'. Which is probably true. If your test is high value, written well, works correctly, then you do get risk mitigation. Your code is more likely to contain bugs if you don't write a high quality test for it. The risk of that happening is greater. But that is only true if we assume that first sentence, that the code coverage is using high quality valuable tests.
Code coverage ensures that critical parts of your application are tested thoroughly... if you're writing valuable tests in the first place.
The bigger issue with the author is that they're making an argument nobody is discussing. Obviously if everything is tested with a quality test, then it'll be thoroughly tested. It's a circular argument.
The question is whether code coverage is good even if the tests suck, or if it's better to not write tests at all than to have bad ones.
18:50 Medical manufacturing equipment doesn’t really test code coverage. Instead they test that the equipment detects a fault for any type of failure. So a sensor staying on or off shouldn’t make every part produced look like a good part. This is tested by running the machine normally while forcing a failure.
This is much closer to that tiger style than testing individual functions because it tests the full behavior in the production operation.
There’s no point in testing if a function throws if it’s just going to be wrapped in a Try Except later.
True test coverage would be not running every line, but running every possible input configuration. A program free of bugs means that all input configurations (values, timings, execution environment/external conditions) result in the expected behavior. Obviously it's not possible to test this exhaustively for most applications and "running every line of code at least one time" is a very flimsy proxy for it.
surprised that TDD isn't represented in the comments: If you follow it rigorously, you only write tests that actually do something. You NEED to write a test that fails, before you write the code to make it pass. this includes compilation failures and empty functions.
You do the call to a non-existent function, you create the function, you do an assert, you write the code. Rinse, Refactor, Repeat
From my experience (15 years in testing) I believe the exact number is 123.456789% -- no more, no less. If you can't get that exact number for divisibility reason, remove or add lines of code as needed.
Even with 100% of unique paths coverage, you can end up with the expected output for the wrong intermediate reasons. Maybe you got to the result via 3rd item in a list when the correct behavior actually would be for the 2nd item in list. AFAIK a "unique path" does not care which of x items in a loop gets you there, both are treated as the same path. "My code is 100% covered." - "Yeah but does it work correctly?" - "I dunno, ship to PROD and let's find out!" :D
I use coverage as an indication if I missed something in my code, but I also ignore/skip lines that are not necessarily need to be tested, like safeguards (like catch exception, write log, then throw it again)
This also reminds me of one time when a coworker created data to be used by the tests and used the random for giving them random names. (above the 2 initial record, using a get_or_create it possibly generates 0 to 20 additional entry depending on if their names match with a previous one or not)
I was thinking heavily how to politely tell them my kind disagreement...
Shouldn’t code coverage only flag logic blocks instead of reporting lines covered? I never understood why I need to test creation of objects since it is just variable assignments for the values passed to it, this is why C# has the ExcludeFromCodeCoverage annotation, because no one is trying to improve the code coverage algorithm.
7:53 why devs dont write tests ?
- frequent changes to features
- not required most of the times
- company gains no $$$
- apart from devs, everyone else sees tests as a lame excuse to not finish on time
- test grow in complexity with regards to the feature
And like irl, you could do everything right and still fail due to factors outside your control. Getting a entry-level job in IT comes to mind.
One of the most useful benefits of testing is that it prevents introducing bugs later on when updating code (regressions)
It's good to have it as A goal, it's just not THE PRIME goal. It's okay to generally look at, especially as a metric over time to see "are we adding a bunch of code without any tests?"
It's still better to have tests than not have any tests.
Quality is better than quantity, but if you can have both it is even better. The problem isn't having a goal of 100%, the problem is prioritizing 100% code coverage over quality coverage.
use coverage to identify what needs testing. Dont write tests simply for the sake of coverage. Quality of tests is still key.
Also quality of coverage. Line coverage is the lowest quality. There are other, better ways to cover code. Branch coverage, toggle coverage, and functional coverage are a thing.
Usually have this kind of argument with my old colleague as well. It's not about 100% or not, but it's about not aiming for it in the first start. 100% may be better than 90%, but trying too hard for it gives off a mentality of it being the "ultimate" goal.
Code coverage ensures all business cases are correct.
If you aren’t testing a branch, it’s either redundant, unnecessary, or a potential bug.
Just delete code till you get 100% coverage. 😅😅
Integration tests, test a portion of a real system and give real signals about whether it works or not :)
Allows you to redesign a project and know for real if it's still working or not during that process and at the end
Unit tests with mocks, not so much, as the real system is only modelled, not used directly
The fundamental problem with the article is already that he doesn't even go into all the different metrics of code coverage and goes straight into LoC coverage which in itself is one of or the most useless metrics to begin with which leaves the impression he doesn't even know what he's talking about in the first place.
Integration test coverage is desirable, and saved my butt multiple times when performing large code migrations. Imo that one should be close to 100%.
12:50 well that one depends, yes if its as simple as that then it doesnt make sense, but if i need to for example check if what coming in is a primenumber, then it makes sense making a function like this, since the amount of code that you need to check if something is a prime number, while not to big, having to hardcode into every function that needs to check if something is a prime number is incredibly annoying to say the least.
The biggest lie is thinking 100% is the maximum value.
Between unit, integration, e2e... that's at 300% for a maximum value.
At the end of the video Prime put the unit tests on top of the pyramid, it should be at the bottom. The width indicates how many tests you should have relative to different layers. The testing pyramid indicates that you should have more unit tests than other types of tests.
I’m sure Prime knows this, it was a little mistake at the end of a session.
I think it's a nice to have, although if you really want 100% coverage you can't just run every line once, you also have to unit test every function. Otherwise you only test one case for every function, which really is not a lot
The whole unit test BS is one of the things that has tainted modern software development. I am sick of seeing zero value tests and none or limit testing on the things which really matter, which won't be obvious and won't be picked up in functional testing but will have dire consequences if its wrong along with not being noticed for along time.
Tests are just as likely to contain bugs lol. They’re good for guarding against regression IMO. And testing things you’re uncertain of.
The first time in my life I look at a Rust project using Tarpaulin having no idea wtf it is, prime drops a video on code coverage, based.
It irks me when people make single statement methods and not make them expression bodied in C#.
I think you misunderstood what they meant at 4:06. To me it seemed like they were arguing that if you aim for 100% then if it isn’t 100% you can see there is some dead code.
For critical software or systems, formal verification and other formal methods may be used to provide stronger guarantees than testing.
My take is that 100% code coverage will never be 100% issue coverage, just a nice way for everyone to pat themselves on the back while they circle-jerk. My "happy place" is : Have UML use case diagrams. Write APPLICATION tests for when the use case should work. Write APPLICATION tests for when the use case should fail. Write APPLICATION tests for when a use case bug shows up so you don't have a regression. Write UNIT tests for that bug "unit code" so that you don't have a regression. No it's not 100%. It tests things that should work do, things that shouldn't work don't and that things you fixed don't get broken again. I'm happy with that.
Coverage without mutation testing (aka testing your tests) means nothing, at least in theory.
With experienced devs that write good tests (and testable code), mutation testing is often overkill. And 100% coverage too by extension.
Unless you write mission critical code, just write useful tests in a reasonable / available time, 60 to 80% branch coverage being an ok (but not mandatory) metric.
I often see and like to see end to end tests covering the "happy" / nominal paths + all specific or error cases covered by unit tests (with maybe some end to end thrown in there, like one to test your errors are correctly transformed into a nice http response etc.), that way you end up with a nice pyramid and pretty quick tests at scale. And if you're motivated, some full integration tests on a live server regularly exectued with smth like playwright, but you may have actual human testers instead.
Also mocks are ok if done right, that's just a lot of work to maintain so don't mock everything. To be able to unit tests without mocking too much you have to stop having a "method call tree" with methods calling methods calling another service methods etc., and try to have as much isolated pure functions not calling anything else as possible (at least not necessary to mock during tests), and prefer composition as early in the tree as possible, hopefully the composition will be covered on the happy path and not mocked.
Like do not code (of course oversimplified, corner and error cases at each level not shown):
[handling service / module / layer / whatever]
handleGetFoosWithBars()
return getAllMappedFoosWithBars()
getAllMappedFoosWithBars()
foosWithBars = getAllFoosWithBars()
return mapToResponseBody(foosWithBars)
[Foo service]
getAllFoosWithBars()
foos = getAllFoos()
bars = getBars(foos)
return merge(foos, bars)
getAllFoos()
foos = getAllFoosFromDB()
/* apply some biz stuff and transform and stuff */
return foos
[Bar service]
getBars(foos)
bars = getBarsFromTheInternet(foos)
/* apply some biz stuff and transform and stuff */
return bars
[Foo repository]
getAllFoosFromDB()
/* sql shenanigans */
[Bar HTTP client]
getBarsFromTheInternet(foos)
/* http shenanigans */
But something more reasonable:
[handling service / module / layer / whatever]
handleGetFoosWithBars()
foos = getAllFoos()
bars = getBars(foos)
foosWithBars = merge(foos, bars)
return mapToResponseBody(foosWithBars)
[Foo service]
getAllFoos()
foos = getAllFoosFromDB()
return pureBizMethod(foos)
pureBizMethod(foos)
/* apply some biz stuff and transform and stuff */
[Bar service]
getBars(foos)
bars = getBarsFromTheInternet(foos)
return pureBizMethod(bars)
pureBizMethod(bars)
/* apply some biz stuff and transform and stuff */
[Foo repository]
getAllFoosFromDB()
/* sql shenanigans */
[Bar HTTP client]
getBarsFromTheInternet(foos)
/* http shenanigans */
Yes, on the whole application you will have duplication. Composition is usually not DRY. But unless your modules / services are overcomplicated, KISS > DRY and composition is more easily testable because it increases the number of methods "on the side of the end to end call flow" that will be pure and depends only on their arguments. It alsos usually produces easier methods to unit test. And it also reduce risks of circular dependencies when doing dependency injection because single responsability and the tree is shorter and wider instead. + your flame graphs are nicer to read :)
100% correct: Code Coverage covers the executed lines and not all scenarios. For example, given a simple function add(a,b) = a + b. One line. How many scenarios given that an and b or 8bit unsigned integers?
Also I condemn the abuse of mocks. I used to work for this company, where the API code base were indoctrinated with CLEAN CODE and were full of mocks and mini files. I didn't know what the point was other than to guarantee the function in its perfectly isolated environment (like in an incubator) was executed... so that such and such CC metrics was met.
I wonder if we just settled on code coverage because that was the 'best' metric that gave any indication that things are tested well. Is there a better metric?
100% means that you tested all cases you are aware of. It doesn't cover things you don't know about.
This article focuses on behavioral-based tests, plus if code was not written to be testable in the first place, people won't write tests.
12;27 earned my like. Yes. There is honestly no reason for that back-asswards pattern.
"ThrowIfLessThanOrEqual" is more characters than the function body lmao
I think code coverage is a good way to push devs into writing tests and kinda ensuring tests won't be an afterthought and won't always come up in reviews as comments saying "hey, we need tests for this". I think it can build up a habit for writing tests, but it also promotes chasing that metric instead of writing proper tests.
Code coverage as a metric of code quality is absolute garbage. I've seen bunch of times where code coverage said that certain lines of code were covered, but they actually weren't working as they should. But hey, management can brag "we have XY% code coverage".
I remember a place doing 100% code coverage. A code review would say "why aren't you at 100%?" and this was a very, very, very, very, very tough section of the code in the sad path. The chance that we'd hit said code would be very hard (though that sentence is a guarantee that someone will hit it the next day).
Other workplace "team, are you okay with us not testing this tough edge case" and team said "ok". The manager then said "the team ok'd it. You're good" I miss that manager. The place's code coverage was very high. The sad path was also very well-coded.
In the words of Rumsfeld: "There are known knowns, there are known unknowns, and there are unknown unknowns". I'd say the first 50% percent of coverage is typically the main good path. The next 25% or so are the known corner cases and foreseeable error conditions. The next 25% percent are impossible error conditions and pedantic checks. But the real danger is lurking in between those two, in the unknown corner cases and unforeseen error conditions. The reality is that increasing code coverage does not help you discover those, and adding tests (often regression tests) for those conditions when they are discovered in the wild (or QA) does not increase coverage, it can even reduce it.
Noob question. How do you test UI stuff? The only way I know how is to run it and see what happens.
How do you 100% cover a lambda.. generated during runtime
I heard AI will give us 125% coverage, by covering the code we are yet to write. ;)
THE ONLY THING I MOCK IS PRIMEAGEN
Every project has a Test environment. Some of them also have a separate Production environment.
Code coverage only covers the code that is present in your project, not the cases that are missing from code.
Devs that don't care about good tests can easily write a test that covers every line of code and then the only check actually made is asserting the behavior of a mock. I've seen it. Still counts towards code coverage. Much more effective for test quality is mutation testing, but it eats up a lot of dev time.
19:03 broo like software for your cars door handle probably has 100% code coverage.
Indeed, code coverage is an useless metric I've never cared about.
I rather prefer employing the idea of 'specification coverage': every single important specification should have corresponding tests.
By doing so, I can confirm that the application behaves correctly or as specified as long as the tests have passed.
Another point is that wiring tests in such way enforces developers to be aware of the behaviors and specifications of the applications.
I skip writing tests for non functional requirements like logging and protective guard clauses.
Note that this is the case for backend dev where application servers never crushes and every runtime errors are handled automatically: responding with 5xx codes while reporting errors to monitoring services like Sentry.
This is the reason why I can skip writing tests for non-functional requirement codes even if they can raise runtime errors.