This is great! Would be great to see a similar video on how to go about understanding the limits(load test) of an event/message based application, if possible
What would be interesting is if you included an example of 'here is an example of an API that where stressing it caused it to fail.' Then apply some fix, run it again, and compare resulting metrics. But great video/idea.
This video is a setup for a future video in which I will use a soak test to identify a memory leak in .NET. Just needed to have this one out first for reference.
We use NBomber. It is C# based (well, there is F# underneath). As I understand, it should have very similar functionality. But it is C#. We were very reluctant to support anything in not-C# and NBomber covers most of our needs. There was a bit of a struggle with implementing load tests. But in 9 cases out of 10 it was because of a lack of performance testers in the project and not a fault of the framework.
@@nickchapsas If I remember right, NBomber was initially an internal project for testing a gambling system. And they have some ridiculous number of request - like thousands per second. I would think that they would want to see pretty consistent results. On our meagre 30-50 request per second, I haven't noticed any significant inconsistencies. Though from the video I see that K6 gives much more detailed result log. In case of our project, we decided that we should stick to the C#. In my experience, it is much easier to get other engineers involved with whatever thing you are doing if it is in the same language as the main project. After working for 5 years on a project where development and autotesting were in two different languages, I would rather fight a herd of radioactive monkeys than support anything on Ruby.-)
I'd love to see load testing integrated on a website. I am building more APIs than I used to but I still manage a good number of websites that don't consume a dedicated API.
Great info! I knew jMeter but I didn't like it too much. Now I can test a logic endpoints sequence in many ways. We'll be implementing this on my work. Thank you!
How do you define a real spike number? I mean looks that you choose a random high number, but how u determine a realistic spike for your scenario. THANKS
For any commercial system, you should ideally have an environment that mirrors production in terms of performance, but has different data stores. e.g. PRE-PRODUCTION. You could configure K6 as part of the deployment pipeline for your PRE-PRODUCTION environment so that you can easily see performance improvements or degradation over time. You could also set up the same thing in your QA pipeline. Although the actual numbers would be different to a production like environment, you could still get a sense of performance degradation or improvement.
Where do you run these tests effectively? Because the machine config might give you different results, right? Does it make sense to run this locally on your laptop?
Can we control rps to be a constant? For example, I want 1500 requests per second to occur for 10minutes, 800 for 5 minutes.. I get that it depends on number of users and sleep, combinations, but not sure how to calculate what numbers I need to setup based on rps.. thank you very much for your time and answer if you have the time to answer!
Question: Do you have to run your test against your CI build? Or your staging? or your prod? How do you compensate the differences in hardware when running test local or in CI?
@@nickchapsas so, you run in your CI against your pre-production? Can the bandwidth of the CI limit the results? I mean, you are "emulating" a lot of different users from a single machine (the CI server)
@@ciberman CI would run TF or Pulimi to provision infrastructure specific for the test and servers to execute the tests against it. The CI agent wouldn't run the tests directly but indirectly
Thanks for the awesome video! I have nodejs backend that is hosted in Cloud Run so should I load test locally or on Cloud Run? Do I have to load test all API routes individually or all as a batch like you did in the first stress test?
Would love to see how to integrate a stress test into the check-in process with multiple devs committing several times a day. Is the pattern to bottleneck on a single deployed environment? Or how do others manage to run tests where code needs to be deployed to validate a pull-request?
I think this is something that would need to run on a nightly schedule and not after each push. That way it does not block the build pipeline and you still get performance results daily. If needed you should also be able to run this type of build manually if there is a need to push something out to production and you want to ensure that there is no performance degradation before releasing any code. It all depends on what you want...
Hello Is it possible to perform correlation using the showcased tool? How can one store a response value from from api call to use in another one? Thanks.
Hi Nick, great video and very interesting topic. If this is part of your pipeline, let's say for production deployment. Do you target production or do you set-up a temporary environment or even a slot to load test against? Running load test on the actual production env could affect current users right? Also I wonder how you deal with authentication using JWT, can you do some automated refresh step? Since the token would expire running those hour long tests.
Hi Nick ... Can you please tell me how to create server job to send email to all users without Api still waiting to complete sending all email ... By the way i am using SMTP from Microsoft account .... Thanks 😊
Hi Nick, I am a performance test engineer I want to know that in Load testing my requirement are put the load around 200 rps not more thant 200rps so is there any technique of throughput controller as we seen in Jmeter or Can I increase more load during test running state like locust tool
great video. One question though: this 'framework' handles just a simple endpoint. How does the project framework look like when we're supposed to handle 10 endpoints? Each endpoint with a separate Load.js, spike.js, etc? Or is it possible to use the same .js file for multiple endpoints; if yes, how does the code look like?Thanks
You can manage that in any way you want. The framework gives you the tools for both one or multiple endpoints either in batch or in sequence and you handle the full flow any way you want
It would be awesome if you show how to test API with xUnit Framework. I know how to test with SQL server but testing with MongoDB in the same way seems impossible.
Thank you for this insanely simple and straight forward way to do load testing.
I would actually watch the jmeter video too btw... I'm so in.
This is great! Would be great to see a similar video on how to go about understanding the limits(load test) of an event/message based application, if possible
Very helpful! I like that Nick shows all these small and big things that sometimes difficult to find somewhere. One of my fav dev channels!
This is a great breakdown of how to get started with K6!
What would be interesting is if you included an example of 'here is an example of an API that where stressing it caused it to fail.' Then apply some fix, run it again, and compare resulting metrics.
But great video/idea.
This video is a setup for a future video in which I will use a soak test to identify a memory leak in .NET. Just needed to have this one out first for reference.
@@nickchapsas makes sense!
Such a good tutorial, leaving a comment for algorithm. Thanks mate
so simply explained great job
Thank you for simple and clear explanation.
Great info, this is something I always intend to learn about but never know where to start. Thanks!
Thanks will check K6. I usually use Jmeter but it is hard to maintain it.
We use NBomber. It is C# based (well, there is F# underneath). As I understand, it should have very similar functionality. But it is C#.
We were very reluctant to support anything in not-C# and NBomber covers most of our needs.
There was a bit of a struggle with implementing load tests. But in 9 cases out of 10 it was because of a lack of performance testers in the project and not a fault of the framework.
NBomber is significantly less consistent in my experience and less performant.
@@nickchapsas If I remember right, NBomber was initially an internal project for testing a gambling system. And they have some ridiculous number of request - like thousands per second. I would think that they would want to see pretty consistent results.
On our meagre 30-50 request per second, I haven't noticed any significant inconsistencies.
Though from the video I see that K6 gives much more detailed result log.
In case of our project, we decided that we should stick to the C#. In my experience, it is much easier to get other engineers involved with whatever thing you are doing if it is in the same language as the main project. After working for 5 years on a project where development and autotesting were in two different languages, I would rather fight a herd of radioactive monkeys than support anything on Ruby.-)
Never knew it was so simple to do, definitely gonna try this.
Yes please the load testing for web apps please Nick!!!
This is great, thanks! I’ll start using it today.
Great video Nick!
Thank you! This was very helpful. I'd love to see these applied in JMeter.
Looks pretty good. Will definitely be testing this out.
Awesome! Really good video and tool.
Very well explained 👏
Hi Nick, thanks for this explanation
Very nice tutorial. Thanks!
This video is just amazing🎉👏
Nice one thanks for sharing ! Yes need k6 on websites as well please. Thank you once again for this video
Nice video!
Tip: you can use CTRL+L instead of clear. Also there is recursive search feature CTR+R
Thank you very much for the excellent explanation!
nice and easy to understand video, pls also do a video with jmeter
nice. waiting for website testing video
Will be nice to see something similar for websites.
Nice one. Exactly what I need to know right now.
Thanks for this amazing video sir!
Thanks man! Thats what I needed.
So cool bro, I have to try it
Superb video !
Nobody likes stress but we can always learn to manage it. Keep it up!
Thanks Nick. Request to share video on mqtt load testing with X.509 certificate for .net.
Great video, I would like to see a video for testing web apps like blazor server
I'd love to see load testing integrated on a website. I am building more APIs than I used to but I still manage a good number of websites that don't consume a dedicated API.
Thanx man that was huge
Great video, I will use it as a starting point for performance testing on my project!!!
Any input about how k6 works with APIs that needs OAuth oidc based authentication will be great
You’d have to manage the flow as part of the test or have a stage where you authenticate the users before you start firing the tests
Very useful, thanks.
Awesome content Master Nick! Perfect
I would just add a POST call to the video
hello nick, love your videos! maybe you can make a video about sonarQube and how it can integrate with your project and tests
One of the Bestttttttt video .
Great info! I knew jMeter but I didn't like it too much. Now I can test a logic endpoints sequence in many ways. We'll be implementing this on my work. Thank you!
yeah!
Thankssss, this is just what i needed, Like and subcribed
Awesome. Can we get on web applications
This is great stuff :)
Fantastic
Pricey test for PROD on any cloud, you know ;)
Please do for websites/web applications also
How do you define a real spike number? I mean looks that you choose a random high number, but how u determine a realistic spike for your scenario. THANKS
Hi Nick, you said you will be doing this type of video for a web application. Can we expect it soon?
In a CD pipeline which environment would you run these tests against? Would you build a production like environment to run them?
For any commercial system, you should ideally have an environment that mirrors production in terms of performance, but has different data stores. e.g. PRE-PRODUCTION. You could configure K6 as part of the deployment pipeline for your PRE-PRODUCTION environment so that you can easily see performance improvements or degradation over time.
You could also set up the same thing in your QA pipeline. Although the actual numbers would be different to a production like environment, you could still get a sense of performance degradation or improvement.
Nice one
Where do you run these tests effectively? Because the machine config might give you different results, right? Does it make sense to run this locally on your laptop?
Can we control rps to be a constant? For example, I want 1500 requests per second to occur for 10minutes, 800 for 5 minutes.. I get that it depends on number of users and sleep, combinations, but not sure how to calculate what numbers I need to setup based on rps.. thank you very much for your time and answer if you have the time to answer!
where can i download your code you just showed? for practice?
So it's similar to that of Jmeter.
Do you have the source code shared somewhere (github or something)? Thanks!
Question: Do you have to run your test against your CI build? Or your staging? or your prod? How do you compensate the differences in hardware when running test local or in CI?
Usually for those tests to tell you something useful, you need to run them in environment that mirror's prod. That is usually pre-prod
@@nickchapsas so, you run in your CI against your pre-production? Can the bandwidth of the CI limit the results? I mean, you are "emulating" a lot of different users from a single machine (the CI server)
@@ciberman CI would run TF or Pulimi to provision infrastructure specific for the test and servers to execute the tests against it. The CI agent wouldn't run the tests directly but indirectly
Please make one for Web with dependency injection please.
I would take a look at codes about entire stress_test. Where do I figure out this code?
can you make a video for load testing for mobile apps like react native apps
You seem to be predicting the problems I’m trying to solve.. lol I wonder where I’m going to find someone top notch when I recruit my dev team 😉
Thanks for the awesome video! I have nodejs backend that is hosted in Cloud Run so should I load test locally or on Cloud Run? Do I have to load test all API routes individually or all as a batch like you did in the first stress test?
How did you plot these graphs ?
how about using 'West Wind WebSurge' tool?
Awesome video dude, keep going, like and subscribe!
very cool, I cannot find your code.
Oops, I thought this was another kind of API ... 😂 I was looking for API (American Petroleum Institute) but got shown this ...
haha it happens!
Would love to see how to integrate a stress test into the check-in process with multiple devs committing several times a day. Is the pattern to bottleneck on a single deployed environment? Or how do others manage to run tests where code needs to be deployed to validate a pull-request?
I think this is something that would need to run on a nightly schedule and not after each push. That way it does not block the build pipeline and you still get performance results daily. If needed you should also be able to run this type of build manually if there is a need to push something out to production and you want to ensure that there is no performance degradation before releasing any code. It all depends on what you want...
How to authenticate the API with bearer token in K6 ..? Need help
Would you still recommend this now it's bought by Grafana?
I’d recommend it even more now
Thank you for the swift reply, I’ve found this video very helpful. Do you have any courses on this?
@@leahsmartie Not yet but I will
Hello
Is it possible to perform correlation using the showcased tool? How can one store a response value from from api call to use in another one? Thanks.
Hi Nick, great video and very interesting topic. If this is part of your pipeline, let's say for production deployment. Do you target production or do you set-up a temporary environment or even a slot to load test against? Running load test on the actual production env could affect current users right?
Also I wonder how you deal with authentication using JWT, can you do some automated refresh step? Since the token would expire running those hour long tests.
can we handle the constant options in the command line for load testing
You can yeah
do a jmeter please...
Link for JMeter video pls... if not pls make it
Hi Nick ... Can you please tell me how to create server job to send email to all users without Api still waiting to complete sending all email ... By the way i am using SMTP from Microsoft account .... Thanks 😊
As far as I know target is virtual users. the question is - is virtual user = ordinar user from real life?
This depends on how you set it up. They can certainly reflect an ordinary user yeah
@@nickchapsas how it can be configured?
Good video but I wish it was more practical (as implied) than theoretical.
Hi Nick, I am a performance test engineer I want to know that in Load testing my requirement are put the load around 200 rps not more thant 200rps so is there any technique of throughput controller as we seen in Jmeter or Can I increase more load during test running state like locust tool
great video. One question though: this 'framework' handles just a simple endpoint. How does the project framework look like when we're supposed to handle 10 endpoints? Each endpoint with a separate Load.js, spike.js, etc? Or is it possible to use the same .js file for multiple endpoints; if yes, how does the code look like?Thanks
You can manage that in any way you want. The framework gives you the tools for both one or multiple endpoints either in batch or in sequence and you handle the full flow any way you want
Where is the code you are using in your videos?
It’s available to my Patreons
It would be awesome if you show how to test API with xUnit Framework. I know how to test with SQL server but testing with MongoDB in the same way seems impossible.
This blows JMeter out of the waters
agreed!
Please can you include the link to the code on your repository
The code is available to my patreons
Thanks for the amazing content Nick, would appreciate it a lot if you could create similar content for website as well.
Running the tests on local machine, won't give real results cause it depends on your computer. What we might do to make the tests more realistic
I never recommended that you run the tests locally. You run the tests against a pre-prod environment from a good server.
👍🏽
Can you share the source code for this?
The source code is available to my patreons
js load tests 🤣🤣🤣
It’s a js engine but the tests are run in Go
Great video Nick!