We have shitty tests that randomly fail and depend on one another. Now we got a perfect framework to hide it and not having to fix these damn tests. YAY!!!
lol recently I started to have some tests (exactly 14 tests) fail the first time I run all tests, but when I rerun them it works. It has something to do with TimeProvider, and the error is "Cannot set the time into the future" or something. I have no idea why they fail because the time provided is hardcoded. Luckily they never fail in the test pipeline
Same, if run in parallel they break. It's not the units, but the tests don't inject dependencies independent from one another. They were written after the code. We'll do it right next project, lol.
Hmmm maybe overtime it will show more advantages over other testing frameworks, but NUnit/XUnit are both decent, well supported, and feature rich, so it does make it more difficult to switch at the moment. But that being said, it's new and no doubt will get better, so who knows 😊
This is pretty cool. If I'm writing unit tests, its true I might not see the value with this framework if all my tests are perfectly isolated. But for integration tests/UI tests, i can finally not have it run in one giant method and finally have test order control! This is really useful for different types of testing.
many of our tests basically do Test1: see if thing 1 works, Test2: see if thing 2 works(which also requires thing 1 to work) Test3: see if thing 3 works(which requires the 2 earlier to work) so the DependsOn is something that I think can be useful. The previous tests checks that the current states are as they should be, so could of course all be put into one test, but having different tests for different purposes is better in my opinion. Like if I want to test that when I start an activity it should have x,y,z and another test checks when the activity is in progress and should then have a,b,c etc
Nunit3 has perfect order control and also you can disable parallelism. Just use [Test, Order(1)] void mytest1() And use [disableParallel] on the test class (you must look up the exact word but there is something like this)
NUnit also has testplatform support. NUnit also has most of the test preparation hooks demonstrated here, and also supports awaitable tasks. So, I think this video is a bit misleading. The most relevant information (and thanks for that!) is that dotnet changed its test execution strategy, and this is actually really useful.
So much hate in these comments. If you’ve never ran into the limitations of xUnit then you wouldn’t realize the benefits. A lot of people override the standard xUnit framework because parallelization in xUnit is bad and not as configurable as it should be. Once you do that it’s extremely difficult to use spec flow on top of it or some other framework This is excellent to see its source generated. It’s the way to go
It is not hate, it is just sad realization how it will be misused in so many ways, making codebase worse. For example, when last time I was asked to check what the problem with test parallelization, it turned out to be not problem in tests, but race condition in code under test. And such things will just mask underlying problems by setting “proper” order and parallelization settings in tests
Thanks Nick. It's interesting to discover new libraries, though I don't see myself using this one before years. The Retry feature is a real plus. When a 2-hours long CI/CD fails for the third time because of flaky unit tests (among ~40,000 unit tests in the solution) you want to see the world burn. All the rest, I don't think I've encountered situations where Xunit was limited.
If you have a flaky code, then you need to fix the code, not adapt the freaking tests, dude. It is not acceptable to have a flaky code in a first place.
I would like to see simpler scaffolding for testing databases and apis, especially when it comes to delays, intermittent connections, corrupted data, partial data, really slow responses, connection strings, security [certs, encryption,etc] and more. While things like an in-memory database are good, it'd be nice to test more realistic scenarios with a project, so you can handle things better.
I love this framework and I love the assertions too, because I like my assertions to both start with "Assert" so that I can scan them in the code easily and also to be fluent. I can proudly say that I am this project's first sponsor!
Note that it’s English fluent. It makes little sense to someone whose english is not their native language. Little of computer science does indeed. So please, stop "fluenting" things just for your side of the Atlantic ocean. Assert.IsEqual(a, b) used to be fine, Assert.That(a).IsEqualTo(b) is nowhere near fluenter. I do love the project tho, and will avocate for its use on next projects.
You got a large amount of files in an assembly. A problem in one file and this thing in an unrelated file causing the issue. Seems like it would be annoying.
Most existing tests I’ve run across just leave the weird behavior in place and let their pipelines fail occasionally. People hate troubleshooting tests because most of these times the issue is in the test code and not the code being tested so they view it as a waste of time. I personally will fix them as I come across but I definitely respect the opinion that if you could just add a depends on attribute to make the order explicitz and quickly fix the issue that might be the worthwhile solution. I’ll still come by and fix it later for you but not everyone needs to be testing experts.
A lot of discussions here about tests should be independent, tests shouldn’t have a complex SetUp, etc. Guys, have you ever seen UI test? It’s a true mess. You ANY-WAY will have such things like depending tests on each other, huge set-ups, both for class and for each test, and yeah, this ObjectBag thing will be useful too. I find this project just perfect for UI testing, very excited to rewrite the entire 8k-tests-framework with it 😅 Ah, dreams… P.S. By the way, parallelism is a must in UI testing too, otherwise you will be waiting results for a couple of days instead of 6-8 hours. So built-in parallelism is a great one, too.
Looks exciting, but I was hoping to see property-based unit testing here. Patiently waiting for the first library that makes it really accessible for .NET.
So after a year after paying for it, I finally got around to doing your unit testing course. I'm loving it and working on implementing it everywhere.. Now you tell me Xunit is obsolete! 🤣
Oh, I see so many ways how this library will be misused by developers. Instead of fixing flaky code that crumbles under tests, it will be just masked by set of attributes that will run tests in very specific order to make them green. “Before assembly” - wonderful way to shoot not only your leg, but also legs of your colleagues, who will guess what is happening there (and it happens somewhere 3 folders away where no one would search for it). Also, not a fan of async assertions. Why? We should already have results by the assertion time. The only thing I like so far - code generation (at least, “on paper” - didn’t try it yet). All other features will be misused way more often than used for good.
Two things: When you do the full video, I would like to see how you would emulation collection fixtures in XUnit. My guess, looking at the preview, is that I would create a base class that implements `BeforeAssembly`, wire up a static property on the base class, then in all my child tests I could access that. (E.g. for an EF Context, some class that wraps API requests, etc.) Second thing, what are your thoughts on creating an abstraction around asserts? Cause it seems like every assert library decides to do things its own way, and if you switch libraries that ends up being the worse part of the migration; changing all the asserts. If you abstracted that out into your own assert classes, and had different concrete implementations for different test libraries you could more easily swap the asserts out by just changing how you implement them in the abstractions. You'd still have to fix the test attributes, but that could probably be done through a mass find/replace.
I must admit, i never used any 3rd party for unit-testing in .NET. So every library is most likely better than the one we use: MS-Test. But T-Unit looks good so far and i really like it too that you have lots of control over it. But don't like the assertions. I would rather have manual Assert.IsTrue, IsFalse, AreEqual, etc. If i want that chaining thing, i would use fluent assertions. But myself would never use that, because i really hate chaining api's in general. I think its a stupid idea and i see no benefits of that pattern at all. It looks nice, writing a one liner but as soon as your line exceeds the editor width or has more than two lines - then it gets less readable and gets extremely hard to debug.
I wonder what would happen if there's a circular dependency of tests. A depends on B. B depends on A. Hopefully a runtime exception. Worst case is a deadlock...
Me personally I like xunit approach, each classes has constructors and disposability or destructors and more natural then attribute mountains and more independent and more threadsafe int that way. But the only benefit is the speed because of the generated code and that's it. I did not like to much yet however can have potential
Having the test result in the Before hook makes so much sense... I can now async wait for the result in the before hook to see the result later, so i can continue overengineering.
2 หลายเดือนก่อน +1
Thanks for the info. As always, refactor all unit tests and use it in production even if it's in preview library, you will thank me later.
Would you test deletion of a record before creating it? Would you test fulfilling an order if the payment gateway returned unsuccessful for payment? With unit testing your assertion stands valid. With integration testing, you are testing the integration of multiple services or scenarios, no matter what.
@@paul.fletcher i agree, my point is the feature "depends on" to make sure that another test ends before.... is just against the rule that a test should be independent
Thanks for sharing this, Nick! Personally, I have mixed feeling about it. I definingly see a lot of benefits it brings for integration tests. For unit tests, the only real benefits are speed and a bit more control on the setup/teardown of the test. I'm a bit worried to open up the test project one day and see a complex graph of test dependencies along with bunch of retries... :/
Great video. Maybe in the next video, go over their benchmarks. Apparently TUnit is 10x faster than Nunit and TUnit_AOT is 75x faster. It would be great if you can verify this and the pros and cons of each method.
The biggest problem I have with test arguments is that if you are using a matrix of test arguments and expected results, most frameworks see that as one test. I'd want to use the matrix of arguments to test the cases, and if one edge case fails, the entire test fails. What would make something like this more ideal for me is that the multiple test arguments are broken out, at least for reporting when there's a failure, as individual test cases. I think the idea of the Before decorator is good, but it would also be great if you could define categories of tests which depend on on or more of those setup methods, so that state might be reused but different fixtures might require different setup states and that could be used to apply what is needed for each test. There's a condition where one test might inherit two different conflicting states, so some thought needs to be put into that type of system, but I see a lot of potential.
You can do [Before every(Class)] which would affect every fixture. But no reason you couldn't have an if statement in there and perform logic based on any information from the context object.
In the UK, it's nUnit preferred over xUnit and MSTest, but I think I will be advocating for TUnit. Might have been good adding TUnit in the title of this video as I watched it thinking it was part of the next update to .NET
DependsOn can be useful in unit testing. If my method has a lot of code, that is covered by multiple tests, then a change that will make a test of a piece of code that is early in the method faile will make a lot of tests fail. If the tests for the later code depend on the test of the early code, then it will be much clearer in the test results where te problem is.
TestContext is such an awesome concept. And no reflection omg, omg omg. Going to buy your testing course. My favorite is still the G-G-G-G-G-Unit. Poppin tham thangs....
Yeah, I expected something really flashy. But aside from using the new test runner, it mostly feels like NUnit. Maybe it has a few more options for "Before", but otherwise, it feels the same.
@@Maloooon The benefit of this new unit testing framework is that it's much faster than both nUnit and xUnit, and supports AoT, which is becoming a bigger and bigger thing in *some* use cases. For th eprojects I'm working on at work it probably wouldn't matter much, but I can see some that could really use those benefits.
@@coloresfelices7 in my opinion assert syntax is weird, shared context is not great, and the way of writing setup and teardown is not as good as just a constructor and a dispose method. Xunit is just a lot less complecated and a lot more barebones and easy to use. Also in my experience xunit is a lot faster too
Personally I think the ConstructorInjection concept is really great and it really shines when you have complex setup/teardowns, because it gives you so much flexibility and control
Am I missing something here or does NUnit + Moq and resharper perfectly sufficient? You can already pass in arguments, you can even specify a nameof and pass in an enumerable etc. I like when we could use Assert.True and Assert.NotNull etc rather than the wordy "Assert.That." syntax
Nick, would you consider making a video about required CET in .NET 9 and the 10-100% method calling performance regressions that come with it on any CPU that isn't a modern intel processor?
"I don't think you understand how great that is of a thing" Well, you're right, I don't. Maybe it's just me, but I don't feel you've explained what's so great about TUnit. Also, since it's source generated, I expected a bit more. For example, the custom properties in TestContext is still a dictionary of objects, so you can't really have a strongly-typed configuration. But compile time errors for [Arguments] are nice.
What's preventing me from using xUnit is the lack of parameterized test classes from NUnit. -It seems like TUnit does not have this.- It seems like you can. The same ArgumentsAttribute in the method can also be used in class constructors.
@@nickchapsas so i can decorate a class with [InlineData(1)] [InlineData(2)] like I could in NUnit using [TestFixture(1)] [TestFixture(2)] and have the tests in a class run with different values in the constructor? Or maybe I'm calling this feature the wrong name. I've seen how people do this in xUnit, and you're supposed to write custom drivers to achieve this.
Really, most projects just need some library that can assert things and some framework that can run those assertions. I've never met a real-life scenario where unit-tests related reflection would be the bottleneck why unit tests run slowly. Also these days you might want to use a library that LLMs know about, so that you could autogenerate test cases sometimes. But that's about it, it's not that complicated.
Tests were supposed to the simple and very less depending on anything else. I do understand the power it brings but it the question is does it really require it?
2 หลายเดือนก่อน +1
The title of the video and the example was a bit confusing to me. My initial reaction was "Is he advocating that we start writing our test code inside the actual classes? Dear lord the mess!". Why would you test an Add method inside a class called Test inside an assembly with the name Tests. All in all, I think a more real world class would have proven a better example of what TUnit is trying to bring to the table, or maybe it would just have made it a bit less confusing to me as to what the big change to testing was/is. I found the documentation for TUnit, but their example was the same code you were showing so that didn't help.
Hmm - I think I'll wait for xUnit v3. xUnit v2 does have major limitations when it comes to integration/acceptance tests that this library does seem to address, but I'm not keen on the level of flexibility this introduces which would potentially lead to some bad practices. xUnit v3 seems to address my issues with v2.
I'm curious how the code generation performance works on large projects. I'd rather suffer through test discovery delays when I'm running tests than suffer through code generation delays every single time I build - but if the code generation/build step isn't particularly noticeable, that could be an interesting tweak.
Heya! Ive got a benchmark on GitHub for building but admittedly it is a very minimal test project. I'd be happy to extend it but writing dummy tests in itself is a bit of effort so isn't the highest on my priority list right now. As for local development, it hopefully shouldn't affect you too much as it uses the newer incremental source generator. That means it only generates new source if it detects changes that would affect it. So hopefully it'll keep performance to an acceptable level!
Source generators don't normally run at compile time. They run every time a file is changed. If you clone a repository, or clean the build artifacts directory, it takes a little time to generate the files, but once that's done they're normal C# files and shouldn't affect the built times much. In theory source generators could cause a lot of work every time you type a character, but incremental source generators do a quick analysis to see if the change would require it to regenerate the source. Most of the time it doesn't. I haven't read the source generators for TUnit, but I suspect it only needs to run the source generator when one of the attributes is changed or added. Tldr; it shouldn't cause much of an increase in build times.
CodeGenerators are incremental, so you would only suffer the upfront cost once and then new tests will just add to the output, rather than trigger a full regeneration. I think it's a really nice idea
I don t buy into stateful unit tests, before after and all that goop. I have a static creating the sut and dependencies, then the test method does arrange act assert. Everything s pure. No state, no hidden stuff no magic. For integration tests I use specflow.
This opens up several windows where we had to create abominable structures. For flakiness, I had to develop a whole retry library... Does this support Specflow?
Nice, it's closer to NUnit rather than XUnit. I like they put `Before(Test)` which is feature I miss from NUnit but not in XUnit. But, I have to await for assertion? Ok, time to get used to.
Required by whom? Nowadays it would make more sense to specify *Sync for the rare synchronous method :). In C#, specifying the return type in the name of a method is superfluous. Better to employ an analyzer to produce a warning in case a Task isn't awaited or assigned to a variable.
I have mixed feelings. While I'm sold on supporting the new test platform, and source generation over reflection. The rest of the features I'm not sure are all good idea. Maybe, maybe not. I'm not passing judgment yet. I do wonder if xUnit could be updated to support test platform and source generation?
Isn't test dependencies are kind of antipattern? Each test should highlight single problem and should be independent and self-sufficient? Also, the "Before" initializers are too implicit in several ways: first of all logic of initialization could be spread among test classes and be "surprising", and second - not very clear what parameters can be passed to "Before" attribute - no code completion with choices can be used. Performance is great btw.
There's analyzers that tell you what parameters to pass to each before/after method. And sharing among multiple classes won't be a thing for before/after class, unless you use before every/after every. I tried to make the language obvious. If you use a normal before/after(class) it'll only affect tests in the class it's defined in
Normally I agree with this, but, I've had a number of cases of end to end certification tests that require work to have been completed on a third party object in a previous test before it can run. Marking those as skipped or failed could save a few minutes per test run in cases were particular integrations are down.
@@ВладиславДараган-ш3ф For unit tests yes. There are other types of testing where shared state can be crucial to testing the thing properly. Its subjective and all depends on what you're testing, how it was designed, what it's doing ,and the level of complexity. Saying that sharing state between any type of testing is pure evil just shows your inexperience. Looking at your previous comments on Nick's videos tells me you're the kind of person who doesn't work well on teams or existing products because you'd spend more time complaining about how bad it is than actually moving it forward.
@@mynameisshadywhat sharing state even in integration tests leads to fragile shitty code that yes, I will complain about lol. Saying that's OK and refusing to isolate tests is a skill issue.
I completely agree that Unit tests being in the same project as the class being tested would be a game changer. After digging into it a bit more I don't think that is how it works in TUnit. Much to my disappointment. It would be so much easier to write and refactor code if the test code didn't have to be in a separate project.
I’m keen to know how people do versioning in .NET if they aren’t using CI/CD. I can’t find a way to do it that’s not manual. I had a way that worked in the project file using dates but that made Nuget packet manager behave strangely.
You can use gitversion and run it in your build script. Which is basically how we do it in our CI which is nothing more than an automated way to run your build script. (just don't tell me you don't use version control either.)
Not a fan of the assertions part at all but I do like the rest. Would prefer to use Shoudly for assertions and have sync or async tests at my own discretion rather than being forced to be async It would be interesting if a test could return something that could be passed to the next test.. but that’s probably too crazy
I see the same problem with TUnit and NUnit, which is not present with XUnit, which is thats it's a lot easier to use and code tests quickly with resharper. With just base resharper if I decide some line in my code needs to be setup before the test just a quick keyboard shortcut and its automatically moved to the constructor. This make writing tests a lot faster and easier since you don't need to write boilerplate code by hand. Also XUnit assert syntax is a lot better in my opinion
Man, I'm retired, but I know somebody is going to have to update thousands of my test cases to this.... once management prioritizes it. Can these be mixed so new tests use this and the old tests can be left in place?
I think it's better to add these futures to XUnit or NUnit instead of recreating a library or maybe a package for XUnit or NUnit to add these futures because most of us don't like to go for a new library
I'm also not really following the power of this tool over other frameworks. In my testing flow i wish there was a simple way to cache the dependencies between tests
I hope it will have better documentation than xUnit... I mean, I hope I wouldn't need to analyze library's source code to find all supported attributes and to understand what they do. That alone makes xUnit very hard to recommend
I like they use NUnit style assertions because you can read like a human. Assert.That(something).IsEqual(thisOtherThing). Why programmers wants to read code when they can read english?
I'm excited about this, especially the direct control over parallelism, but I'll be sticking with Shouldly for my assertions. I can't stand Overly.Fluent.Interfaces.That.Create.A.Class.For.Every.Word.In.An.Assertion. If your framework has a class called "Is" or "It" and it's NOT a mocking library, you've made a wrong turn, and you need to reevaluate your decisions. "But it's so discoverable", you say. Great, that helps me for the first ten minutes of using the framework, and then it's a speed bump for the rest of my career. Just stop it. Give me a "ShouldBeLessThanOrEqualTo". Oh look, I just "discovered" the method I wanted in ONE auto-complete step instead of seven, and I only had to type ".sblto" in the IDE. Velocity matters.
Integration testing is different. You're testing suites of behavior. 1) Create a new user. 2) Retrieve the just-created user. 3) Update their email address. 4) Check that the change was persisted. 5) Clean up after yourself by deleting the user. 6) Check that the user was deleted. These are six different tests that have to run in a specific order. Yes, you can cram all of these into a single "CRUD" test, but the granularity of 6 individual tests is better for visibility. Now, what I need is the ability to say that #3 depends on #2, and #4 depends on #3, but #5 should only run after #3 & #4 even if they fail. I want to know what the "CRD" parts work, but the "U" is currently broken.
@@MelGrubb Awesome so you can make 6 bad interrelated tests instead of 6 independent ones. And for some reason feel good about it because they're integration tests, not unit tests
@@IronDoctorChris Integration tests are sometimes about entire workflows. This, then this, then this. It's not all the time, certainly, but I definitely write workflow/scenario-driven tests on occasion. It would be nice to have a testing framework that supports this by design rather than me having to write one multi-step test. If you don't like it, or can't wrap your head around complex testing scenarios, then nobody is trying to force you. I'm just saying built-in support for those of us that can would be nice.
@@IronDoctorChris I'm with you on this. I would question why @MelGrubb is splitting these out. The overarching business requirement (which should be documented in a user story as an acceptance criteria) is that changing an email on a newly created user is persisted. My thinking would tell me that is a single use-case/workflow and should be represented in a single integration test. The moment you start splitting this out, you end up with all kinds of problems with dependencies between tests. My observation in all my years of automated testing is that a large number of developers don't seem to understand how to write tests that are self contained and don't interfere with other tests. This becomes particularly apparent when multiple tests are run in parallel, so I've seen lots of organisations mandate 1 test run at a time! The solution here is not to have the library provide a band-aid (allow dependency setting) but to write all tests properly in the first place!
Personally I prefer PUnit.
Testing in Production :3
🤣🤣🤣
@@IsaacOjeda The Crowdstrike method
This comment made my day thank you
Are you a Microsoftie?
yeah, it the best. cause user-generated scenarios
We have shitty tests that randomly fail and depend on one another. Now we got a perfect framework to hide it and not having to fix these damn tests. YAY!!!
lol recently I started to have some tests (exactly 14 tests) fail the first time I run all tests, but when I rerun them it works. It has something to do with TimeProvider, and the error is "Cannot set the time into the future" or something. I have no idea why they fail because the time provided is hardcoded. Luckily they never fail in the test pipeline
Same, if run in parallel they break. It's not the units, but the tests don't inject dependencies independent from one another. They were written after the code.
We'll do it right next project, lol.
@@fredrikjosefsson3373 Do your tests read the appsettings.json file? If I may ask.
Now I can over-engineer my tests easier than ever! ;)
Hmmm maybe overtime it will show more advantages over other testing frameworks, but NUnit/XUnit are both decent, well supported, and feature rich, so it does make it more difficult to switch at the moment. But that being said, it's new and no doubt will get better, so who knows 😊
This is pretty cool. If I'm writing unit tests, its true I might not see the value with this framework if all my tests are perfectly isolated. But for integration tests/UI tests, i can finally not have it run in one giant method and finally have test order control! This is really useful for different types of testing.
even with unit you have less configuration to be used. You don't need to inherit from shit, you have it as a more direct approach.
many of our tests basically do Test1: see if thing 1 works, Test2: see if thing 2 works(which also requires thing 1 to work) Test3: see if thing 3 works(which requires the 2 earlier to work) so the DependsOn is something that I think can be useful. The previous tests checks that the current states are as they should be, so could of course all be put into one test, but having different tests for different purposes is better in my opinion. Like if I want to test that when I start an activity it should have x,y,z and another test checks when the activity is in progress and should then have a,b,c etc
Nunit3 has perfect order control and also you can disable parallelism.
Just use
[Test, Order(1)]
void mytest1()
And use [disableParallel] on the test class (you must look up the exact word but there is something like this)
NUnit also has testplatform support. NUnit also has most of the test preparation hooks demonstrated here, and also supports awaitable tasks. So, I think this video is a bit misleading. The most relevant information (and thanks for that!) is that dotnet changed its test execution strategy, and this is actually really useful.
Liked for "Assert DOT THAT DOT IS DOT " -Dance
You mean this th-cam.com/users/clipUgkx-4EdhRx-UT3dlAvXrm1yaPxkCOGV74EQ?si=gFuOk8PJd8Nnx8E1
@nickchapsas can you provide fluent examples that you prefer?
So much hate in these comments. If you’ve never ran into the limitations of xUnit then you wouldn’t realize the benefits. A lot of people override the standard xUnit framework because parallelization in xUnit is bad and not as configurable as it should be. Once you do that it’s extremely difficult to use spec flow on top of it or some other framework
This is excellent to see its source generated. It’s the way to go
If you have parallelisation issues with unit tests you definitely have bigger problem than xUnit
@@ВладиславДараган-ш3ф this
1. I am wondering is SpecFlow dead and Reqnroll is the replacement
2. How long until we have support for other frameworks - i.e. Reqnroll?
If you're using spec flow you have bigger issues than xUnit
It is not hate, it is just sad realization how it will be misused in so many ways, making codebase worse.
For example, when last time I was asked to check what the problem with test parallelization, it turned out to be not problem in tests, but race condition in code under test.
And such things will just mask underlying problems by setting “proper” order and parallelization settings in tests
This gets me excited.
I'd love to see more stuff about parallel execution of tests and how this stuff works with Azure pipelines.
Oh hell yes! I likey!!
Yeah, this should get some funding sent it's way for sure.
Thanks Nick. It's interesting to discover new libraries, though I don't see myself using this one before years.
The Retry feature is a real plus. When a 2-hours long CI/CD fails for the third time because of flaky unit tests (among ~40,000 unit tests in the solution) you want to see the world burn.
All the rest, I don't think I've encountered situations where Xunit was limited.
If you have a flaky code, then you need to fix the code, not adapt the freaking tests, dude.
It is not acceptable to have a flaky code in a first place.
Nunit also has retry since a long time
I would like to see simpler scaffolding for testing databases and apis, especially when it comes to delays, intermittent connections, corrupted data, partial data, really slow responses, connection strings, security [certs, encryption,etc] and more. While things like an in-memory database are good, it'd be nice to test more realistic scenarios with a project, so you can handle things better.
I love this framework and I love the assertions too, because I like my assertions to both start with "Assert" so that I can scan them in the code easily and also to be fluent. I can proudly say that I am this project's first sponsor!
All they did is copy java
@@hellowill Grow up
Note that it’s English fluent.
It makes little sense to someone whose english is not their native language.
Little of computer science does indeed. So please, stop "fluenting" things just for your side of the Atlantic ocean.
Assert.IsEqual(a, b) used to be fine, Assert.That(a).IsEqualTo(b) is nowhere near fluenter.
I do love the project tho, and will avocate for its use on next projects.
Before assembly, what a fantastic way to hide weird behaviour in a large test suite.
Because doing things before tests is weird. And how is it going to hide it if the code that runs is annotated with Before(Assembly)?
Not true for acceptance tests where you’re spinning up a bit suite of tests.
You got a large amount of files in an assembly. A problem in one file and this thing in an unrelated file causing the issue. Seems like it would be annoying.
Most existing tests I’ve run across just leave the weird behavior in place and let their pipelines fail occasionally. People hate troubleshooting tests because most of these times the issue is in the test code and not the code being tested so they view it as a waste of time. I personally will fix them as I come across but I definitely respect the opinion that if you could just add a depends on attribute to make the order explicitz and quickly fix the issue that might be the worthwhile solution. I’ll still come by and fix it later for you but not everyone needs to be testing experts.
One usage where this can be beneficial is running your tests against an in memory server. You can start it before your tests, and dispose it after.
DependsOn is pretty much what I've wanted for a number of my integration tests. Thanks for sharing!
Are there any other unit testing libraries that use the new testing platform?
I think NUnit has support for it
2:15 I love how nick chooses "random" numbers
If only the test ran for 260 seconds too
What about support for build pipelines? Are there any pros and cons there?
Use dotnet test in ur pipelines
This is awesome as we don't need vstest host anymore and I personally hate it.
The Arguments attribute was implemented in MbUnit almost 20 years ago. I loved that testing framework.
A lot of discussions here about tests should be independent, tests shouldn’t have a complex SetUp, etc. Guys, have you ever seen UI test? It’s a true mess. You ANY-WAY will have such things like depending tests on each other, huge set-ups, both for class and for each test, and yeah, this ObjectBag thing will be useful too.
I find this project just perfect for UI testing, very excited to rewrite the entire 8k-tests-framework with it 😅
Ah, dreams…
P.S.
By the way, parallelism is a must in UI testing too, otherwise you will be waiting results for a couple of days instead of 6-8 hours. So built-in parallelism is a great one, too.
Looks exciting, but I was hoping to see property-based unit testing here. Patiently waiting for the first library that makes it really accessible for .NET.
What's wrong with CsCheck?
FsCheck is not accessible?
We need *test* frameworks not just *unit* test frameworks (we use them for more than just unit tests)
So after a year after paying for it, I finally got around to doing your unit testing course. I'm loving it and working on implementing it everywhere..
Now you tell me Xunit is obsolete! 🤣
18 Sept, version = 0.1.752.
21 Sept, version = 0.1.793
Pumping out the updates
Oh, I see so many ways how this library will be misused by developers. Instead of fixing flaky code that crumbles under tests, it will be just masked by set of attributes that will run tests in very specific order to make them green. “Before assembly” - wonderful way to shoot not only your leg, but also legs of your colleagues, who will guess what is happening there (and it happens somewhere 3 folders away where no one would search for it).
Also, not a fan of async assertions. Why? We should already have results by the assertion time.
The only thing I like so far - code generation (at least, “on paper” - didn’t try it yet). All other features will be misused way more often than used for good.
Two things: When you do the full video, I would like to see how you would emulation collection fixtures in XUnit. My guess, looking at the preview, is that I would create a base class that implements `BeforeAssembly`, wire up a static property on the base class, then in all my child tests I could access that. (E.g. for an EF Context, some class that wraps API requests, etc.)
Second thing, what are your thoughts on creating an abstraction around asserts? Cause it seems like every assert library decides to do things its own way, and if you switch libraries that ends up being the worse part of the migration; changing all the asserts. If you abstracted that out into your own assert classes, and had different concrete implementations for different test libraries you could more easily swap the asserts out by just changing how you implement them in the abstractions. You'd still have to fix the test attributes, but that could probably be done through a mass find/replace.
I must admit, i never used any 3rd party for unit-testing in .NET. So every library is most likely better than the one we use: MS-Test. But T-Unit looks good so far and i really like it too that you have lots of control over it. But don't like the assertions. I would rather have manual Assert.IsTrue, IsFalse, AreEqual, etc. If i want that chaining thing, i would use fluent assertions. But myself would never use that, because i really hate chaining api's in general. I think its a stupid idea and i see no benefits of that pattern at all. It looks nice, writing a one liner but as soon as your line exceeds the editor width or has more than two lines - then it gets less readable and gets extremely hard to debug.
Thanks for the feedback. You're not tied to the TUnit assertions and can use whichever library you like!
Agree, i use an old version of NUnit just to keep Assert.True and Assert.AreEqual etc
Really need a blazor testing video
There is no AI in it? Pfffff, what a waste. [ɯsɐɔɹɐs].
AIUnit next
He makes a good point chat gpt might not know how to write my tests in it. Yet.
That sarcasm tag is brilliant! Don't know why but I've never seen it written like that before
I wonder what would happen if there's a circular dependency of tests. A depends on B. B depends on A. Hopefully a runtime exception. Worst case is a deadlock...
Hey @JustArion there's a compile time analyzer and also a runtime exception (for if you'd disabled analyzers)
Me personally I like xunit approach, each classes has constructors and disposability or destructors and more natural then attribute mountains and more independent and more threadsafe int that way. But the only benefit is the speed because of the generated code and that's it. I did not like to much yet however can have potential
You can use constructors and dispose in TUnit as well, IIRC
Having the test result in the Before hook makes so much sense... I can now async wait for the result in the before hook to see the result later, so i can continue overengineering.
Thanks for the info. As always, refactor all unit tests and use it in production even if it's in preview library, you will thank me later.
really cool feature. bout to start migrating code to TUnit now. XD
how could ever a test depends on another test ? in my humble opinion, tests should be indepedent no matter what.
Would you test deletion of a record before creating it? Would you test fulfilling an order if the payment gateway returned unsuccessful for payment? With unit testing your assertion stands valid. With integration testing, you are testing the integration of multiple services or scenarios, no matter what.
@@bitmanagent67 in this type of situation, it's the "arrange" portion of the test that should set the table. Not the result of another test
@@carlitoz450 sounds like semantics to me.
@@paul.fletcher i agree, my point is the feature "depends on" to make sure that another test ends before.... is just against the rule that a test should be independent
Thanks for sharing this, Nick! Personally, I have mixed feeling about it. I definingly see a lot of benefits it brings for integration tests. For unit tests, the only real benefits are speed and a bit more control on the setup/teardown of the test. I'm a bit worried to open up the test project one day and see a complex graph of test dependencies along with bunch of retries... :/
Great video. Maybe in the next video, go over their benchmarks. Apparently TUnit is 10x faster than Nunit and TUnit_AOT is 75x faster. It would be great if you can verify this and the pros and cons of each method.
Thanks. What is the best free mocking framework without limitation like in mocking static class or extension method?
The biggest problem I have with test arguments is that if you are using a matrix of test arguments and expected results, most frameworks see that as one test. I'd want to use the matrix of arguments to test the cases, and if one edge case fails, the entire test fails. What would make something like this more ideal for me is that the multiple test arguments are broken out, at least for reporting when there's a failure, as individual test cases.
I think the idea of the Before decorator is good, but it would also be great if you could define categories of tests which depend on on or more of those setup methods, so that state might be reused but different fixtures might require different setup states and that could be used to apply what is needed for each test. There's a condition where one test might inherit two different conflicting states, so some thought needs to be put into that type of system, but I see a lot of potential.
You can do [Before every(Class)] which would affect every fixture. But no reason you couldn't have an if statement in there and perform logic based on any information from the context object.
In the UK, it's nUnit preferred over xUnit and MSTest, but I think I will be advocating for TUnit. Might have been good adding TUnit in the title of this video as I watched it thinking it was part of the next update to .NET
Completely untrue. I’ve worked at ASOS and checkout.com and they all use xUnit. Same goes for Just eat, clear bank and many others
Did you ever use Machine.Specifications? I loved it. Made so much sense.
Great video. The before all and before each reminds me about Pester which is a PowerShell test tool.
DependsOn can be useful in unit testing. If my method has a lot of code, that is covered by multiple tests, then a change that will make a test of a piece of code that is early in the method faile will make a lot of tests fail. If the tests for the later code depend on the test of the early code, then it will be much clearer in the test results where te problem is.
TestContext is such an awesome concept. And no reflection omg, omg omg. Going to buy your testing course.
My favorite is still the G-G-G-G-G-Unit. Poppin tham thangs....
I'm happy with NUnit
Yeah, I expected something really flashy. But aside from using the new test runner, it mostly feels like NUnit. Maybe it has a few more options for "Before", but otherwise, it feels the same.
xunit is streets a head of nunit
@@Maloooon The benefit of this new unit testing framework is that it's much faster than both nUnit and xUnit, and supports AoT, which is becoming a bigger and bigger thing in *some* use cases.
For th eprojects I'm working on at work it probably wouldn't matter much, but I can see some that could really use those benefits.
@@Domse65 Why do you think so?
@@coloresfelices7 in my opinion assert syntax is weird, shared context is not great, and the way of writing setup and teardown is not as good as just a constructor and a dispose method. Xunit is just a lot less complecated and a lot more barebones and easy to use. Also in my experience xunit is a lot faster too
XUnit and NUnit both work very well in F#.
If TUnit is using source generators I am guessing it would be C# olnly.
All that hype in the beginning for a more granular [SetUp] attribute o_O?
Personally I think the ConstructorInjection concept is really great and it really shines when you have complex setup/teardowns, because it gives you so much flexibility and control
Am I missing something here or does NUnit + Moq and resharper perfectly sufficient? You can already pass in arguments, you can even specify a nameof and pass in an enumerable etc.
I like when we could use Assert.True and Assert.NotNull etc rather than the wordy "Assert.That." syntax
For simple UnitTests anything will work, but TUnit really shines in complex integration/acceptance test contexts
I use GUnit
TestsSucksUnit =)
Assert.That(x).IsEqualTo(y) is actually the same approach as in the Java library Assert4J
Would be possible to do one to see how test coverage looks like with this new lib
Nick, would you consider making a video about required CET in .NET 9 and the 10-100% method calling performance regressions that come with it on any CPU that isn't a modern intel processor?
Wait that’s a thing??
"I don't think you understand how great that is of a thing"
Well, you're right, I don't. Maybe it's just me, but I don't feel you've explained what's so great about TUnit.
Also, since it's source generated, I expected a bit more. For example, the custom properties in TestContext is still a dictionary of objects, so you can't really have a strongly-typed configuration.
But compile time errors for [Arguments] are nice.
What's preventing me from using xUnit is the lack of parameterized test classes from NUnit. -It seems like TUnit does not have this.-
It seems like you can. The same ArgumentsAttribute in the method can also be used in class constructors.
xUnit has parameterized test classes
@@nickchapsas so i can decorate a class with [InlineData(1)] [InlineData(2)] like I could in NUnit using [TestFixture(1)] [TestFixture(2)] and have the tests in a class run with different values in the constructor? Or maybe I'm calling this feature the wrong name.
I've seen how people do this in xUnit, and you're supposed to write custom drivers to achieve this.
Really, most projects just need some library that can assert things and some framework that can run those assertions. I've never met a real-life scenario where unit-tests related reflection would be the bottleneck why unit tests run slowly.
Also these days you might want to use a library that LLMs know about, so that you could autogenerate test cases sometimes. But that's about it, it's not that complicated.
Will I be testing this test framework or is it stable to use?
Ah good old Tom :) great engineer back in ASOS days!
Tests were supposed to the simple and very less depending on anything else. I do understand the power it brings but it the question is does it really require it?
The title of the video and the example was a bit confusing to me. My initial reaction was "Is he advocating that we start writing our test code inside the actual classes? Dear lord the mess!". Why would you test an Add method inside a class called Test inside an assembly with the name Tests.
All in all, I think a more real world class would have proven a better example of what TUnit is trying to bring to the table, or maybe it would just have made it a bit less confusing to me as to what the big change to testing was/is.
I found the documentation for TUnit, but their example was the same code you were showing so that didn't help.
Example is the same because probably it’s a paid ad ;)
Hmm - I think I'll wait for xUnit v3. xUnit v2 does have major limitations when it comes to integration/acceptance tests that this library does seem to address, but I'm not keen on the level of flexibility this introduces which would potentially lead to some bad practices. xUnit v3 seems to address my issues with v2.
100k, you deserve it !
I'm curious how the code generation performance works on large projects. I'd rather suffer through test discovery delays when I'm running tests than suffer through code generation delays every single time I build - but if the code generation/build step isn't particularly noticeable, that could be an interesting tweak.
Heya! Ive got a benchmark on GitHub for building but admittedly it is a very minimal test project. I'd be happy to extend it but writing dummy tests in itself is a bit of effort so isn't the highest on my priority list right now.
As for local development, it hopefully shouldn't affect you too much as it uses the newer incremental source generator. That means it only generates new source if it detects changes that would affect it. So hopefully it'll keep performance to an acceptable level!
Source generators don't normally run at compile time. They run every time a file is changed. If you clone a repository, or clean the build artifacts directory, it takes a little time to generate the files, but once that's done they're normal C# files and shouldn't affect the built times much.
In theory source generators could cause a lot of work every time you type a character, but incremental source generators do a quick analysis to see if the change would require it to regenerate the source. Most of the time it doesn't.
I haven't read the source generators for TUnit, but I suspect it only needs to run the source generator when one of the attributes is changed or added.
Tldr; it shouldn't cause much of an increase in build times.
CodeGenerators are incremental, so you would only suffer the upfront cost once and then new tests will just add to the output, rather than trigger a full regeneration. I think it's a really nice idea
hi, nick. is it good to use net 9 rc for production?
what Editor are u using??? Looks like Visual Code but not this
JetBrains Rider
@@restveggie6155 +++
I like the Injectable Class Data Source
How quick is the source generation? Is it noticable?
Does it compatible with AutoFixture?
I don t buy into stateful unit tests, before after and all that goop. I have a static creating the sut and dependencies, then the test method does arrange act assert. Everything s pure. No state, no hidden stuff no magic. For integration tests I use specflow.
Looks like the main release version of Visual Studio 2022 is not ready for prime time yet for TUnit. I couldn't make it work.
This opens up several windows where we had to create abominable structures. For flakiness, I had to develop a whole retry library...
Does this support Specflow?
Nice, it's closer to NUnit rather than XUnit. I like they put `Before(Test)` which is feature I miss from NUnit but not in XUnit. But, I have to await for assertion? Ok, time to get used to.
Good library. Can you compare it wth a new xUnit v3? They also made a lot of changes, comparing to the v2
Isn't *Async suffix is required for async function in notation?
Required by whom? Nowadays it would make more sense to specify *Sync for the rare synchronous method :). In C#, specifying the return type in the name of a method is superfluous. Better to employ an analyzer to produce a warning in case a Task isn't awaited or assigned to a variable.
@@positronalpha It is definition of notation - so would be nice to have.
I have mixed feelings.
While I'm sold on supporting the new test platform, and source generation over reflection. The rest of the features I'm not sure are all good idea. Maybe, maybe not. I'm not passing judgment yet. I do wonder if xUnit could be updated to support test platform and source generation?
Isn't test dependencies are kind of antipattern? Each test should highlight single problem and should be independent and self-sufficient? Also, the "Before" initializers are too implicit in several ways: first of all logic of initialization could be spread among test classes and be "surprising", and second - not very clear what parameters can be passed to "Before" attribute - no code completion with choices can be used.
Performance is great btw.
There's analyzers that tell you what parameters to pass to each before/after method.
And sharing among multiple classes won't be a thing for before/after class, unless you use before every/after every. I tried to make the language obvious. If you use a normal before/after(class) it'll only affect tests in the class it's defined in
Looks good, but not good enough to get me to switch any current projects over. Maybe try it for a greenfield project.
reminds me of the spec flow library
Aren't tests that depend on other tests typically an anti-pattern? I don't see why I'd want to use the DependsOn attribute.
Integration testing or where state matters. Less relevant for unit tests.
@@Thomhurstsharing state between test runs is pure evil
Normally I agree with this, but, I've had a number of cases of end to end certification tests that require work to have been completed on a third party object in a previous test before it can run. Marking those as skipped or failed could save a few minutes per test run in cases were particular integrations are down.
@@ВладиславДараган-ш3ф For unit tests yes. There are other types of testing where shared state can be crucial to testing the thing properly. Its subjective and all depends on what you're testing, how it was designed, what it's doing ,and the level of complexity. Saying that sharing state between any type of testing is pure evil just shows your inexperience.
Looking at your previous comments on Nick's videos tells me you're the kind of person who doesn't work well on teams or existing products because you'd spend more time complaining about how bad it is than actually moving it forward.
@@mynameisshadywhat sharing state even in integration tests leads to fragile shitty code that yes, I will complain about lol. Saying that's OK and refusing to isolate tests is a skill issue.
I like it. I hate tests being a different project
I completely agree that Unit tests being in the same project as the class being tested would be a game changer.
After digging into it a bit more I don't think that is how it works in TUnit. Much to my disappointment.
It would be so much easier to write and refactor code if the test code didn't have to be in a separate project.
I’m keen to know how people do versioning in .NET if they aren’t using CI/CD. I can’t find a way to do it that’s not manual. I had a way that worked in the project file using dates but that made Nuget packet manager behave strangely.
You can use gitversion and run it in your build script. Which is basically how we do it in our CI which is nothing more than an automated way to run your build script.
(just don't tell me you don't use version control either.)
@@pilotboba Thanks, I’ll look into that.
Not a fan of the assertions part at all but I do like the rest. Would prefer to use Shoudly for assertions and have sync or async tests at my own discretion rather than being forced to be async
It would be interesting if a test could return something that could be passed to the next test.. but that’s probably too crazy
Tests can retrieve other tests context objects. And you can store stuff in your contexts "ObjectBag" so that's possible!
I see the same problem with TUnit and NUnit, which is not present with XUnit, which is thats it's a lot easier to use and code tests quickly with resharper. With just base resharper if I decide some line in my code needs to be setup before the test just a quick keyboard shortcut and its automatically moved to the constructor. This make writing tests a lot faster and easier since you don't need to write boilerplate code by hand.
Also XUnit assert syntax is a lot better in my opinion
You can do the same thing in TUnit
Man, I'm retired, but I know somebody is going to have to update thousands of my test cases to this.... once management prioritizes it. Can these be mixed so new tests use this and the old tests can be left in place?
would love to see you do a video putting o1-preview to the test
I think it's better to add these futures to XUnit or NUnit instead of recreating a library or maybe a package for XUnit or NUnit to add these futures because most of us don't like to go for a new library
I would like to see a video about Mocking frameworks now Moq has gone out of fashion. What is your approach to Mocking Nick?
This testing ordering sounds cool but it will lead to maintainance issue eventually. Tests should not depends on execution order.
I'm also not really following the power of this tool over other frameworks. In my testing flow i wish there was a simple way to cache the dependencies between tests
Playwright?
I hope it will have better documentation than xUnit... I mean, I hope I wouldn't need to analyze library's source code to find all supported attributes and to understand what they do. That alone makes xUnit very hard to recommend
I like they use NUnit style assertions because you can read like a human. Assert.That(something).IsEqual(thisOtherThing). Why programmers wants to read code when they can read english?
I'm excited about this, especially the direct control over parallelism, but I'll be sticking with Shouldly for my assertions. I can't stand Overly.Fluent.Interfaces.That.Create.A.Class.For.Every.Word.In.An.Assertion. If your framework has a class called "Is" or "It" and it's NOT a mocking library, you've made a wrong turn, and you need to reevaluate your decisions. "But it's so discoverable", you say. Great, that helps me for the first ten minutes of using the framework, and then it's a speed bump for the rest of my career. Just stop it. Give me a "ShouldBeLessThanOrEqualTo". Oh look, I just "discovered" the method I wanted in ONE auto-complete step instead of seven, and I only had to type ".sblto" in the IDE. Velocity matters.
Thanks for intreducing a new tool to our toolbelt, now we know it exists we can decide if and when it might solve us a problem. Cool library thanks.
What is the point of DependsOnI(OtherTest) ? This is violation of FIRST principles in unit testing. Unit tests shouldn't depend on one another.
Like I said, it’s not for unit testing
Integration testing is different. You're testing suites of behavior. 1) Create a new user. 2) Retrieve the just-created user. 3) Update their email address. 4) Check that the change was persisted. 5) Clean up after yourself by deleting the user. 6) Check that the user was deleted.
These are six different tests that have to run in a specific order. Yes, you can cram all of these into a single "CRUD" test, but the granularity of 6 individual tests is better for visibility. Now, what I need is the ability to say that #3 depends on #2, and #4 depends on #3, but #5 should only run after #3 & #4 even if they fail. I want to know what the "CRD" parts work, but the "U" is currently broken.
@@MelGrubb
Awesome so you can make 6 bad interrelated tests instead of 6 independent ones. And for some reason feel good about it because they're integration tests, not unit tests
@@IronDoctorChris Integration tests are sometimes about entire workflows. This, then this, then this. It's not all the time, certainly, but I definitely write workflow/scenario-driven tests on occasion. It would be nice to have a testing framework that supports this by design rather than me having to write one multi-step test. If you don't like it, or can't wrap your head around complex testing scenarios, then nobody is trying to force you. I'm just saying built-in support for those of us that can would be nice.
@@IronDoctorChris I'm with you on this. I would question why @MelGrubb is splitting these out. The overarching business requirement (which should be documented in a user story as an acceptance criteria) is that changing an email on a newly created user is persisted. My thinking would tell me that is a single use-case/workflow and should be represented in a single integration test. The moment you start splitting this out, you end up with all kinds of problems with dependencies between tests.
My observation in all my years of automated testing is that a large number of developers don't seem to understand how to write tests that are self contained and don't interfere with other tests. This becomes particularly apparent when multiple tests are run in parallel, so I've seen lots of organisations mandate 1 test run at a time!
The solution here is not to have the library provide a band-aid (allow dependency setting) but to write all tests properly in the first place!
Seems they were heavily inspired by Specflow...
Nice
Source generated test library 👀
People still write tests? AI does mine for me and these tests have helped me catch many subtle bugs.
await in a test method for non async code is just odd