When To Unit, E2E, And Integration Test
ฝัง
- เผยแพร่เมื่อ 24 มิ.ย. 2023
- Recorded live on twitch, GET IN
/ theprimeagen
Author (who doesn't follow me): / htmx_org
Grug: grugbrain.dev
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact - วิทยาศาสตร์และเทคโนโลยี
I have never seen prime read an article with as little reading error, I think he really mentally vibes with this kind of writing
I think it's because it forces anyone reading this to slow down and read twice
I truly vibed here as well!
This observation is important to Prime lore
While you wouldn't use a 1TB file for testing you might still process files that large in production, in which case the implementation that tries to load the entire file into memory before passing it to the function that does the work is not going to work well, so you'll probably want to process the data as a stream. But that still doesn't mean you need to open the file within that function, it can take a generic stream object, so in production you can pass a file-stream that reads from your 1TB file and for testing you can pass a memory-stream that just reads some test data from memory.
I was about to write the same exact comment :)
Testing the wrong things or testing for now
Yeah, but in that case Stream would be an abstraction, FileStream the real implementation and MemoryStream the mock implementation. It's just that in this case they already are provided to you by the environment. But replace this with some remote service, external device reader etc. and that's when you have to create mocks.
... and then you've a test case that never fails a read. You still need cases that exercises all paths. So, you'd need a separate function wrapping any file call that can fail. Bam, you're now mocking io fails and 1 function is now six functions.
Sending file contents is a ticking bomb. You pass the buck for the error conditions, but you've a function that's blind when the world around you changes (as Carmack puts it). You need something that's testing the size of that input and warning when it goes beyond expected usage. That's logging now instead of unit testing. The buck has been passed again to the production side.
Me see grub, me put like
I prefer grub2
Me see grub, me boot Linux
your description of seperating data sources and the actual code under test (passing the file content) did really help me write better code and have more fun writing tests.
The last place I worked had flakey tests in all of their services. Plus tests were not run in CI even though CI existed. Everyone looked at me like I was the crazy person when I made a large effort to debug some of the failing tests. Even better most of the tests were unit tests with huge mocks 🙃. Let's just say I got out of there very fast
A unit test with a mock is most often useless. As you are making assumptions as to how that foreign device or system work. And you’re developing something that’s relatively complex that you can’t test. The only mocks that are reliable are those that the manufacturer made and maintains. I’ve had a situation where a customer used mocks and simulators and their production release crashes. Because they assumed (wrongly) that the send() would always send the whole packet. Which is the case on a localhost and fast Ethernet but it doesn’t over the real internet. So the first chairs they wanted to book at the airline force the system 😂
One look at the code and I laughed so hard, “TDD is. It replacement for sloppy ignorant developers”. Even without any automated tests this would not have happened in my case, because I always check every results for production code.
I fixed it without even running the useless test, I just booked a ticket and canceled it again. “That’s is the only real test! End to end baby! Now automate that and forget those useless unit tests” 😂
We did this for energy- and medical systems which need to be certified every time you change something. So we never trusted on unit test only integration and E2E. The units you’ll test anyway by hand when you develop. And the when you write proper code, you’ll don’t have to change them again. Because you never change an individual lego brick. You’ll make a new one but never change the old when it does what it was specified to do.
In the last place where I worked as CTO, I found we had over 1000 tests, but for each run a number of tests was failing, randomly. It took a long time to fix the broken tests, established the causes, and eliminate the useless ones, to have 100% pass. Broken tests, especially random failing tests, are useless, they make the entire test suite a waste. Also most of unit tests are a waste. People who do write tons of unit tests are noobs following trends, like the ones using TDD.
@@biomorphic I like having you as a CTO!
I agree that 99% of the unit tests are utterly useless. You test those visually when you write them and when your units are extremely small you never need to touch them again. Any discrepancies don’t show up in the unit tests anyways because a strict unit test doesn’t have connection to external resources and that’s where the most of the exceptions come from.
And that argument that when an integration or a e2e test fails that you don’t know what causes it, shows two potentials issues
1) your changes are too big and you need to commit smaller and more frequently.
2) your code base isn’t properly structured.
Because what an engineer does is actually analyze where a cause is. A car mechanic doesn’t run unit tests he runs in modern cars a series of integration tests.But great mechanics just hear it feel the behavior and know what’s wrong or at least where to look.
I started my career as an EE and fixed hardware. You don’t even have schematics (akin to source code) and you just had to figure it out. And we did most of the time and that’s stuff we didn’t even design ourselves.
@@biomorphic "most of unit tests are a waste"
It really depends on the projects.
@requizm do you need to write a unit test for passing a string into a function that is typed as an object when typescript or compilation would literally prevent this from happening? I see unit tests like this all the time probably just to push numbers up by creating fake tests that don't actually test anything. College kids are taught to "unit test every singe possible way you can expect a function to fail" and it leads to a lot of nonsense.
The 1TB file guy works at Tom's startup.
Tom is a genius, his git repo has only one json of 1kb and 1terabyte git history
Never test until you know what the program output should be. If you know what the output should be, you can write a test. You should write that test as soon as you know the output, no sooner, no later. Don't add more than one test at a time. If a test you didn't just add is failing, it's a regression. Keep your testing simple or it will summon the shadow demon.
I decreased my reliance on integration tests and write mostly unit tests. Increased coverage, decreased test time, eliminated flakes tests because only integration tests introduce flakeyness.
Writing a million mocks that’s necessary to make integration tests is so painful and can take more time than it took to write the feature I’m in trying to test. I just use integration tests for main scenarios and handle all my edge case testing in unit tests.
Focusing more on unit tests also has the effect of encouraging more procedural programming rather than OOP, as you’ll stop overloading classes with private methods and start moving logic into static library functions that can be unit tested.
i usually make a test whenever i notice a spot where a certain level of complexity passes through. it's great intuition that saved me a lot of time. the more serious tests come later when the code feels more solid like mentioned in the video.
I find it funny how we came from "we hate system modelling" to "oh no i dont know how to unit test before developing because we dont have system modelling"
(its not a critique, I really find it funny because it happens to me too)
Also this guy's way of writing is so fun lol and hell yeah that mocking problem \m/
Everyone hates planning ahead before they dig themselves in a hole that basically requires rewriting the whole thing to get out of.
3:47 grug use unit test, cage complexity spirit demon
Using mostly e2e and never have problems with Grugs kind of mystery breaking tests.
Many years ago the team were into unit testing but we found those rarely saved us. The problems were often in steps between the units.
Worked with TDD fans for a while, it can work without too much messing around if you just assert html containing text. But stopped doing it, I could never really feel at ease with thinking a failing test was fine and normal.
That comment about not mocking was gold to me, as right before you said that I was thinking, "But how can you not mock? I don't want to use the real thing in tests." And yet there was a third way: factor out the dependency so that to write a unit test for that function amounts to testing a third party tool, so you wouldn't, and then you have the complex function that just needs a simpler input that's easy to construct in a test.
The place I work at now as a paid part-time intern, the first real task given to me when switching to the backend was to write mock tests to increase branch coverage for the whole repo to 80%. The only tests we have now are some sort of integration/system test based on a few hundred megabyte large files. Soo, two paid interns are now tasked with writing tests with mocking for the whole codebase. It's the most painful experience of my life.
Every article should be written in grug
Time to start the Rewrite it in Grug movement
@@JonathonMcClungBrb gonna implement gruglang
I could get on board all three of these ideas.
One of my coworkers, the most senior of senior devs, "fixed" a flickering test. His changes broke a bunch of other tests that now randomly fail. He doesn't seem to care, so I take that as I can ignore them aswell.
just ask him and have a chat about it
@@lukasmolcic5143 The problem is he is basically omnipotent, and trying to talk to him would be more trouble than it's worth. I'm just there for the paycheck at this point until I find a new job.
Sooooo good! :D @2:07...calling out Dave Farley -- glad someone is doing it! and in such a humorous, balanced and accurate manner! ...soooo good! really enjoying this!
I urged Dave Farley to stop promoting the mockist unit tests many times and he just don't get it how bad they are. He told me that a huge system he was programming and is full of mocked tests is very successful...
I like TDD for Go w/ Testify, personally. Often I'm using "golden" tests as my starting point though because I can define my expected inputs and outputs. Then, I write more verbose tests on bits that I'm not as confident in. I find that I always end up wasting time tracking down bugs when I get lazy and skip unit tests to try to get it done
Mixing TDD with FP is my approach, it helps me break down problems into smaller pieces that I can easily wrap my head around.
Tests also double up as documentation, like google maps for new team members.
Congrats on 100K!
My experience with test first has changed how I write code. My code is more reliable and my tests are easy to change when business requirements change. The secret? I focus on business value and not on implementation. Also, I use dockerized databases for my testing so I never have to mock repos. It isn’t difficult but it seems that way to those who haven’t done it before.
Edit: fixed a typo
Added note, I’m not saying that I’m more talented than others. My team and I have figured out how to use BDD to guide our development. We believe in BDD and TLD. Everything is just a tool to help you write code. Use what works for you.
dockerized databases are a game changer!
I would like to know more about this. How do you prepare data in the docker db? Do you insert it via sql on start? Isnt it hard to maintain and understand the test data later?
need more details
@@bezi2949 sorry that I'm just now getting to this. When you use a tool like spec flow to implement bdd, you get access to something called .feature files. You're allowed to write tests in the gherkin syntax and you can set up quite a bit of seed data in the feature file directly or in a CSV that is imported into one of the steps, given, when, then. The point is that you can just use your base crud functionality to save all of this data. As your test is running. I like to run the tests, seed. The data in the given statements, run the when which is the action step, run the then which is the assert step, and then delete any data that I no longer want to use for future tests.
@@bezi2949 One more note, you shouldn't need to understand the data later because it's all about testing now. All testing data is invalid after the tests have run so it should be deleted.
Turns off alerts so we don't have to hear "welcome to Costco, I love you", proceeds to say "welcome to Costco, I love you". 🤔
+1 on breaking up IO and logic for testability
Disagree about mocking effects (e.g. file system access, database access, http requests). Testing effectful code is the whole motivation for mocking. Instead of actually performing side effects in your unit tests, you tell the test what you expect to receive from the external system, and then assert that your program returns the expected result. While it's a best practice to extract effects from otherwise pure code and test the pure functions independently, it's also important to test the operations that perform effects (e.g. read X from DB => perform HTTP request with Y=> store response Z in DB).
That sounds like a e2e test which by definition should use mocked data unless you're a real cowboy and decide to use production data.
@@kevinb1594 I would use a staging/beta environment for an E2E test. The advantage of this kind of unit test is that you can handle all possible cases. Whereas E2E cases can handle a happy path + a few common failure modes.
Never in my 2 jobs have we had failing tests across several projects, or any tests whatsoever 😂
11:05 are there more resources to get primes point on this one? i don't think i fully got his example
I stopped doing unit tests (there are exceptions, but generally don’t see the use of them) because my units are so small it’s just useless. It’s like testing to see if you can stick a key in a lock. Of course that still works after manually testing it the first time - I can pin a lock and cut a key! I am more interested if the engine turns over when I turn the key and that all cylinders are firing aka Integration tests. I don’t need to write a unit test to see a spark on every spark plug. We put coils around the spark plug wires to monitor it always (integration, end to end and in operation tests -- that is useful). I know if I get a pulse when that the spark plug sparks.
And then I want to be sure that my truck can still pull the tiles out of the pavement after I tinkered with the engine aka end to end test. I’ll actually put it on a test bench and push it.
The only unit tests I make is for business logic where corporate and governmental inconsistencies make weird exceptions in basic knowledge. And you want to test I’d the input triggers the right logic. I have no comparison in the car world for that 😂
But in medical systems you have a lot of exceptions based on sex, weight, age - and a combination of those, those combinations you need to test in a unit test. But we also twat those in E2E test cases. Especially if they could lead to harm or death when done wrong.
moral of the story - don't reach for the testes until you're fully firm
I like this a lot. I agree, only unit test where the algorithm is complex enough to need it. I prefer API testing with API between components. Test each component/layer.
When Grug use TDD Grug leave club and use metal tools
I think it might still be useful to have one dedicated test to verify your app / tool / whatever doesn't just crash on your 1TB file if that is something your tool is expected to handle, but don't mix that "perf test" with testing functionality. And it is probably streaming data inside, doing some parsing shit, not just holding all of it in memory lol
Grug is a smart grug. Use club to smash like.
There's a certain point where I start to naturally "test" my code to make sure it does what I think it does. You write a little code to make sure it works, and that's that. I usually just throw that code away when I'm done with it. But I think this is the precise moment when unit tests start to make sense. I'm literally already writing them, might as well keep them around.
The whole writing tests before you write code thing never really made sense to me though. Unless you have a crystal clear specification for how something should perform, writing tests beforehand seems quite counterproductive.
Writing temporary tests while you were running your code is part of REPL/interactive development. You could save some of your interactions to serve as examples, and maybe as tests. Writing your tests beforehand as part of and along with behavioral constraints) is part of example or spec or constraint based coding where you give your specs to the computer and a smart compiler may give you a good implementation, intergration tests and end-to-end tests and unit tests are good to give to chatGPT so that it can program more effectively and serve as good comments/documentation that can be partially run.
"If you have to mock you did something wrong". React components testing would like to have a word.
We have so called time bombs. These tests would run normal all the time, but then out of a sudden they fail because of some datetime things.
One test used to fail every day between 0-1pm 🤷♂️
6:54: No failing tests, because no tests exist. But we have a ton of notices, that get triggered per request in various places where "undefined" array indexes, variables and Constants are accessed the wrong way. (Lession for life: "Don't let JS Developers write PHP without supervision")
Most of these missing vars and consts are global, meaning, if we would set or define them, there is a chance that all the parts that do the checks correctly would break.
But no Client ever would pay us for rewriting and testing half the codebase for all 100 client configurations, so we have to use the "Homer Simpson Solution" instead.
Suppress the error output and disable error-logging. (Named after an Episode of the Simpsons where Homer "Fixes" a warning light in his car by putting a sticker over it.)
And before anyone says anything. Yes we actually do fix the notices that we come across while doing other work.
Our Boss just does not allow us to fix the system outside of tasks that are payed by some Client. (Time and Money are very important to him)
Im on the build team. Devs hate me, managers love me because i used the iron rod of god where if your PR fails any unit tests, it doesnt get to be in master. Simple as that. Now our unit tests work and no one ignores failing tests
Why do devs hate you? We shouldn’t want bad code in prod lol
Because many devs only care about writing code and not good code
@@Ivcota You would be surprised. A lot of devs hate even running their sql before committing it.
Grug speaks how prime reads normal english. I'm happy that Prime found a writer that he can read well 😁 (all jokes I love Prime).
you didn't forget, yet we heard the "costo" line from you anyway.
It's so satisfying to have all of your initial code failing because it is a bunch of unit tests and the manager asks how's it going.
People are missing out on the true power of TDD and BDD. You are supposed to spend time thinking about and writing the tests. This guides how your code is written in a much cleaner way than simply writing the code. Of course, write the tests from the view point of the business value being added and not from the viewpoint of your state design pattern.
I completely agree with that, I wanted to write it myself.
I also agree with you here, and feel people in the video totally miss the point. TDD is NOT a testing practice, is a sofware development practice where the tests are a byproduct. It helps you to avoid exactly what they are saying in the video is something 'inevitable', which is, coupling the test with the implementation. When you do it and implementation changes, all your tests are inmediatly invalid
CONGRATULATIONS
Not sure what the aversion to mocking is about. Even in the fs example you provided, your file read function would still need try-catches or other branching logic to validate the file was successfully read. And then at that point you need a way to emulate what happens if the file is not found or some other error occurs (hence the need for a mock). Mocking is just a natural extension of dependency injection.
post the whole thing as 1 video at this point T_T
Why? This format is much better for appeasing the YT-Algorithm-Gods. A playlist would make more sense.
@@AScribblingTurtle True, it might be better for the algorithm but when I watched the first video I liked the article he was reading but it stopped in between which was kinda disappointing. Then the second one came and I was like oh nice but again it gets divided into a 3rd one and idk if this is the last one.
Congratz!!!!
I think I have grug brain, can't quite understand (8:50) what does Prime mean by bad data/flaky test.
Also, thank you Prime for the demo 10:30, really useful for someone that hasn't built things with automated testing and is trying to learn more about it.
:) ty
A flaky test is one which is non-deterministic. You can run it 100 times in a row without changing a single line of code and sometimes it will pass, other times it will fail.
This can happen quite a lot: if your tests use randomized test data, or reference the system time, or have async code in them that could have race conditions, or if your test runner runs the tests in a non-deterministic order and you are sharing some data between tests, etc etc.
Flakiness can happen with any kind of test: E2E tests, integration tests, or even unit tests.
"Keep It Simple Stupid" 😂😂😂
9:27 he’ll yeah!! No mocks!!
He remembered!
10:47 Prime know PureFunctions without knowing what it is called.
any source recommendation for testing ?
grug is a genius
Grug seems experienced :)
"I'm a tool and I build tools ..." hahahaha
Doing the whole "never integrate with your environment thing" is really really hard though. Very often you need little pieces of data that you do not know at the start of the function so you cannot really extract it out entirely. Then you can either mock part of your data access layer or run a local db or similar for the test. I sometimes prefer the latter, but the downside is that your tests that could have taken milliseconds now take seconds to complete.
How do you not know what the data should look like?
Are you just jumping right in to implementing without having a definition of done pinned down?
@AlexanderNecheff It´s not about not knowing the structure of the objects. More that you often have a loop that does xyz for a bunch of items doing one or more queries for each of them. When you have it as an input you have to select all the data at once or you cannot extract the db calls. If you do a lot of data shoveling on bigger datasets that gets unmanageable. You can´t just load 10gb of data into memory. You can say screw that i´m just gonna test it for a single item and assume the rest of the code is correct. It´s either that, mocking or running a db and injecting data for tests. You need a construct like streams to be able to put the data as an argument without actually putting the data as an argument (you did not read the file yet) for that to be viable for bigger sets of data. This very quickly becomes not straight forward if you are not working with file input or stdin but a one or more database and database like stores.
I have only used mocking when working with a flaky 3rd party API that required consumers to build controllers, payloads, and issue requests. There were so many points of failure in implementing their code that mocking had to be the only right solution.
What is the difference between unit tests and integration tests?
Can someone get this guy some more coffee?
After “I hate UML” I thought that there would be a video “tests are not needed”
10:48
Michael Scott
Welcome to Costco. I love you!!!!!!
I have a sneaky suspicion that Primeagen is Grug, Grug is Primeagen... heh!
I am a little confused about the “you shouldn’t mock” premise
Suppose I have an API call and a function that does something with its response somewhere, how can I test that function without mocking the API call? Or did I miss the point completely?
Its just like reading the file example. Don't read the api inside the function that also does the business logic. Separate the Read api step, pass the contents to the logic function. You'll have to put these two together somewhere but this way you can test all logic with zero mocking
The problem with mocks is that you break one of the most important principles of testing: don't mess around with the implementation details. When you mock, you literally replace/monkeypatch implementation details of your code. You are actively making your test more brittle. And it's just laziness too, because all mocking can be avoided by instead refactoring your code and allowing for better dependency injection, like prime demonstrated.
@@kasper_573I agree with the dependency injection approach. However, most JS developers I've encountered prefer just module mocking rather than DI.
@@darrorpsk6148 I still don’t get it, how could I avoid making a real api call in tests without mocking :/
@@pmatari guess if it mocks its an integration test then and the point is "never mock in unit tests"? i have no idea though
mocking sucks, but you have little options with 3rd party code if it doesnt implement dependency injection (which can also become tedious if its overdone, but it sure beats mocking)
His definition of "inbetween tests” is still a unit test. Use these as your unit tests, and TDD is easy. More micro level tests are pointless, as these higher level tests will fail. It took me 15 years to begin understanding this.
I see your testing pyramid and raise you one testing cone.
young grug love wise grug advice
Mocking bad. Fire pretty.
Agreed
Unit testing ought to use for testing the behavior not implementation.
Why the hell would you make a function for a function call?
Its so true, the word 'integration test' causes so many sour faces.
What is your github?
If your complex function needs a 1 TB file, then have it take a stream instead of a 1 TB string, or make it work with partial data and loop it. It's really not hard.
I miss alerts
Is truth. Is good.
bro you don't read in files all at once and then pass around all of the bytes, you read the stuff in small chunks (like 8 KB chunks or so) and then process it bit by bit.
Talking about unit test? It's not Dave Farley, since he hates unit tests basically. :)
Flakey tests are both extra work and usually no value
I have been in a large team with thousands of E2E UI tests. For which one team their tests where flaky. After investigating found out that the testing API they wrote was flaky and once I fixed it concluded that their production code was just as flaky.
Making me learn that flaky tests are good data about who don't know what they are doing. Somehow these people had the arrogance to claim to be the only one to know how to fix the flaky testing issue. Just switch to a different testing framework and only writing a tenth of the total amount of tests.
Just got here
flakey e2e tests arent the worst. At least if they fail enough times in a row you actually know something is wrong.
I miss the notifications
Grub like
TDD works
I love mocking.
"if you have to mock you've done something wrong" and "dont like flaky tests".. well how do you do an integration test decoupled from services you don't control? Surprise, you mock them.
Property testing is not used nearly enough. I think a lot of less experienced devs might not even know that quickcheck libraries are a thing that exists.
dependency injection.
I literally cant even accept anyones negative opinion on tests because i know they're not doing them right in the first place. The testing experts actually acknowledge that you cant test everything in every possible direction, but most people test nothing and when they do write tests they write them after they write the code which is effectively impossible to get right.
Not a TDDer here, but in my understanding, unit tests ARE supposed to be deleted if your API or functionality changes. At that point you are not offering that service, so why test it?
People should rather learn that most things don't need unit tests. You only test a total of 3 things:
1. Any incoming query - assert it returns what it should.
2. Any incoming command - assert it changes what it should, given it's somehow publicly accessible.
3. Any outgoing command - mock to see if it gets sent with the correct arguments.
That's all. You don't go unit testing implementation, cuz grug club you in face.
Mocks are shit, totally agree, are not representative at all
Ok, first. Like me, after I edit to some shit:)
Ooh repost?
Yea get used to it, he has 3 channels and he posts the same thing like twice on each channel
I literally do not. This is not a repost.
Never seen a primeagen repost
Yeah my bad!! I was delirious at 1am, 10 seconds in, and was thrown by seeing Grug again. Thought the Grug video might have been deleted and rereleased. Grug 1 was one of your funniest btw!
Didn't mean to be rude 😊. Sorry if I was offensive.
Disliked since I didn't get to hear "Welcome to Cosco, I love you"
test
answering the title: never, never and never. What's the next question?
Mocks are proof that OO is bollocks
Montana L
? What???
Please translate
Stop unit testing things that are not units.
This reeks of people that doesn't know what the definition of tests are. There are books like CD, Devops Handbook and TDD, Also the Art of Unit testting etc.. The problem in the industry is that junior developers just pull things out of their ass and then try to sell it has law. Before making videos like this update yourselfs first and actually read what the thought leaders have been saying for tyhe past 30 years.