From my tests, the reason why the timing measurements are never exact is because the 1st request is always treated separately, adding 3 extra seconds. So on video's minute 19:23, instead of the expected 18 seconds you get 21 seconds @DanVega.
Thanks for the great content. It seems to be that the comparison is not 100% fair as you reduced from 200 default to 10 threads. So I guess the real difference is probably when you really hit the wall for the performance of the thread per request model. So I guess the more interesting would be to leave the default 200 threads setting and then spin up -n 1000 with less blocking time to show the real power. Nevertheless, I think the the point you showed is great and very thankful. Thanks for very useful content!
In addition to removing the upper limit related to number of cores virtual threads also use less memory. 200 system threads will require around 200 MB of RAM
It should be more about memory. If built correctly, where you needed to scale 4 Microservices for some task, you can potentially get by w/ 2 etc as you're using less memory (assuming the underlying thing you're calling can handle it as well).
Thanks for the wonderful Contents! I was just wondering: are virtual threads a replacement for async programming? I'm slightly confused after going through multiple videos
Hi there. Great content! I really appreciate it. I wanted to ask why we have significantly more failed requests when using virtual threads. I’ve also tried benchmarking in my local environment, and I get the same results. What could be the reason.
Hello, mister Dan. That was a concise and easy to understand overview. But what is the trade-off behind enabling virtual threads? There definitely is a reason why this neat feature is not enabled by default. Should we all just switch out blocking MVC apps to virtual threads and live happily or there must be some consideration?
Code that uses synchronized blocks can pin the carrier thread which can cause performance issues.. some libraries have made changes to their code to be vt friendly, for example, postgresql jdbc connector. On the other side, if you're using mysql, mysql connector hasn't changed their synchronized code so I wouldn't recommend activate them in projects using mysql. So, for now, it depends on your codebase, at some point in the future, the jvm team will change the way synchronized code works so the switch will be easier.
I like it, but migrating a jdk 17 app running in Docker requires jumping through a few hoops to enable jdk 21. Not insurmountable, but more than a single line in a config file. All the same, I love the direction this is going. Thanks for the introduction.
I am curious about thos hoops too. You can you Buildpacks to automatically build the image from source and that comes ootb with the Spring Boot plugin, that makes it actually 0 lines of code.
@jamesStansell and @abelsalgadoromero4402, As with *any* migration/upgrade it will take some effort. First there is the compatibility roulette, of finding what versions of the various plugins you are using work with JDK 21, and which will need to be upgraded. I then also found that for some, they were still under milestones, so I needed to bring in the spring-milestones repository. Finally, if you're just getting going with containers, figuring out how to upgrade the jdk in your container may take a bit of doing too. As I said, not insurmountable, and the move to virtual threads is really great news.
Thank you very much it was very interesting and helpful. just want to add why there is an additional seconds in those tests, need to do warm up to the jvm and after that the benchmark will get more realistic and consistent.
Thank you for the video Dan! I have question about the virtual threads. At 19:03 I see "Failed requests: 5", and after you enabled the virtual threads at 20:53 I see "Failed requests: 52". Does this mean that most of the requests failed?
Great explanation! Should come really handy in IO bound apps. Would this break Spring Security or other things that use ThreadLocal - for example when storing the SecurityContext?
With the introduction of virtual thread, several interfaces have been redesigned. As one of them, Thread Local can also be used in the traditional way. (I think it's the strength of language that runs on vm.) Instead, excessive use of Thread Local can overrun memory, and in fact, it eliminated a lot of use of TheadLocal in the implementation on the java.base package.
Hey Dan, why you didn't limit platform thread for virtual thread testing on properties file as you did without virtual thread? So that we could understand that 10 platform thread concurrently handle 60 request without blocking themselves. Great tutorial ❤
Great! but what is the best practice is i want to enable Virtual Threads for single route in my application ? manually create and run virtual Thread in controller?
You don't enable virtual threads for a single endpoint you do it for your entire application. You don't need to manually create a virtual thread in your controller as Spring is handling that for you.
Expected you to show the final console output to make sure not all 10 platform threads were taken. Does someone know what thread names would be printed?
I don't think @19:27, there was a need to comment out the server.tomcat.threads.max=10 property before running the example with virtual threads Actually, keeping the max tomcat threads same would've validated the non-blocking nature of virtual threads. That is what I got when I executed virtual threads example with max tomcat threads as 10. Same result i.e. around 12sec
Great question. If you were only reaching for reactive programming as a scalability mechanism you now have an option on the imperative side of the house. Reactive programming is still a great architecture choice for streams based processing and dealing with back pressure. Another JEP in Project Loom (Structured Concurrency) is still in preview that is something else Webflux solves for.
Ah right, I forgot about back pressure. So @@DanVega from your experience, what are some common problems that can be solved easily using reactive programming and that get complicated when doing imperative? I am thinking about maybe batch jobs, but even spring has Spring Batch right? I don't think it follows a reactive approach.
Hi Dan, was the max thread limit to 10 only to simulate a hardware (cpu core count) bottleneck? bcz I believe even if you didn't enable the virtual threads, the whole request would take similar time since your machine is capable of it even without the virtual threads. Am I correct?
I think this depends on the type of the pool type. In case of fixed threadpool here matching with core count, for platform threads was meant to dictate don't create new thread for new request/task, beyond 10. Fixed thread pool has the strategy that till the cap is hit it will spawn new thread, unlike fork join pool. Usually setting more than 10 would pretty much be insignificant. After certain point the performance can vary because of context switches.
Do keep in mind that there is a limit to physical threads you can setup. The default with tomcat is 200 - so while this example did constraint it down to 10 artificially - the same example when scaled would produce comparable results. Virtual Thread can effectively work with no limits (or really high) so they have a lot more scaling than physical threads
Not sure why Dan changed max threads property. I have tested similar code with 300 concurrent threads, when virtual threads are turned on, the app becomes unresponsive. Without virtual threads, it just works fine.
Nice video, it would be cool if you help us understand if a billion request come in at once, there will be a memory issue. Can you force the server to only accept so many active request? I always wondered about that.
There are a few approaches here: 1. Do you want to serve all of those Billion requests? Then you need horizontal scaling, with your app servers sitting behind a load balancer, and preferably having auto-scaling enabled. If there's a DB involved, then you will also need to have multiple read-replicas set up, and a lot of caching between your app and the DB. 2. Do you want to discard requests that you can't serve? In that case, you can set up rate limiting. Any requests that breach your rate limit can be served with an HTTP 429 response (too many requests). They client would have to retry the request once traffic has died down. Of course, there are many more ways of handling a billion requests, I've just mentioned these two from top of my head.
Hey Dan - Nicely done on the video! Can't wait to try this out myself. Did you look at JVM memory consumption? I'm curious to see what, if any impact on memory there is. Thanks!
Good to hear from you friend 🤩 I haven't done any comparisons with JVM memory consumption but I like where your heads at. I will take a look into that.
@@DanVega - Let me know what you find. I'm curious from a operation cost perspective. ie running ~20 Spring Boot microservices in a K8 cluster. You can chew up a lot of memory quickly, which can drive up costs due to runtime memory needs.
Thank you for the video. I am building the url with query params.If pass the header then not able to pass the query params. If pass the query params then not able to pass the bearer token. How to pass the bearer token in the header?
From my tests, the reason why the timing measurements are never exact is because the 1st request is always treated separately, adding 3 extra seconds. So on video's minute 19:23, instead of the expected 18 seconds you get 21 seconds @DanVega.
Thank you so much, this is the best explanation on simulating virtual thread.
bro just wanted to say that you're amazing and the things i learned on this channel is the absolute best, thanks to you king!!
That is a great tutorial! Very easy-to-understand examples and it clarified all the important concepts!
Great example! That extra bit of time is most prob due to the OS and some other overhead contending for share of the 10 threads. Pretty impressive.
Thanks for the great content.
It seems to be that the comparison is not 100% fair as you reduced from 200 default to 10 threads. So I guess the real difference is probably when you really hit the wall for the performance of the thread per request model. So I guess the more interesting would be to leave the default 200 threads setting and then spin up -n 1000 with less blocking time to show the real power.
Nevertheless, I think the the point you showed is great and very thankful. Thanks for very useful content!
In addition to removing the upper limit related to number of cores virtual threads also use less memory. 200 system threads will require around 200 MB of RAM
It should be more about memory. If built correctly, where you needed to scale 4 Microservices for some task, you can potentially get by w/ 2 etc as you're using less memory (assuming the underlying thing you're calling can handle it as well).
Thanks for your great explanation, very easy to understand
Thanks for the wonderful Contents! I was just wondering: are virtual threads a replacement for async programming? I'm slightly confused after going through multiple videos
As always best explanation. keep it up Dan!
Nice explanation! Why did we turn off the tomcat.threads.max when we turned the virtual threads on?
It actually didn't matter...its either or, once you enable virtual threads that is no longer relevant. Nice catch
It is very important and cool feature provided and very easy to plugIn. Superb explaination and demo @DanVega. Thank you so much.
You're very welcome!
Great video! Thank you!
Hi there. Great content! I really appreciate it. I wanted to ask why we have significantly more failed requests when using virtual threads. I’ve also tried benchmarking in my local environment, and I get the same results. What could be the reason.
Hello, mister Dan. That was a concise and easy to understand overview. But what is the trade-off behind enabling virtual threads? There definitely is a reason why this neat feature is not enabled by default. Should we all just switch out blocking MVC apps to virtual threads and live happily or there must be some consideration?
Code that uses synchronized blocks can pin the carrier thread which can cause performance issues.. some libraries have made changes to their code to be vt friendly, for example, postgresql jdbc connector. On the other side, if you're using mysql, mysql connector hasn't changed their synchronized code so I wouldn't recommend activate them in projects using mysql. So, for now, it depends on your codebase, at some point in the future, the jvm team will change the way synchronized code works so the switch will be easier.
I like it, but migrating a jdk 17 app running in Docker requires jumping through a few hoops to enable jdk 21. Not insurmountable, but more than a single line in a config file.
All the same, I love the direction this is going. Thanks for the introduction.
Can you say more about what those hoops are? I expect to migrate some workloads to containers next year, likely without Java 21 at first.
I am curious about thos hoops too. You can you Buildpacks to automatically build the image from source and that comes ootb with the Spring Boot plugin, that makes it actually 0 lines of code.
@jamesStansell and @abelsalgadoromero4402,
As with *any* migration/upgrade it will take some effort. First there is the compatibility roulette, of finding what versions of the various plugins you are using work with JDK 21, and which will need to be upgraded. I then also found that for some, they were still under milestones, so I needed to bring in the spring-milestones repository. Finally, if you're just getting going with containers, figuring out how to upgrade the jdk in your container may take a bit of doing too.
As I said, not insurmountable, and the move to virtual threads is really great news.
Thank you very much it was very interesting and helpful. just want to add why there is an additional seconds in those tests, need to do warm up to the jvm and after that the benchmark will get more realistic and consistent.
Thank you for the video Dan! I have question about the virtual threads. At 19:03 I see "Failed requests: 5", and after you enabled the virtual threads at 20:53 I see "Failed requests: 52". Does this mean that most of the requests failed?
We can observe that they're failed req.s because of length mismatches, you can pass -l arg to ab to avoid. So we can ignore them in our case.
Thank you for that answer!
Thank you!@@nm69
Thank you very much Dan, i wanted to ask about failed requests 52, is it right?
hey Dan, thanks for sharing! Being part of java 21 I could assume this is production ready right? great content!
Great content with an excellent explanation, keep going Dan ;)
Glad I'm able to help out
Great explanation! Should come really handy in IO bound apps. Would this break Spring Security or other things that use ThreadLocal - for example when storing the SecurityContext?
With the introduction of virtual thread, several interfaces have been redesigned. As one of them, Thread Local can also be used in the traditional way. (I think it's the strength of language that runs on vm.) Instead, excessive use of Thread Local can overrun memory, and in fact, it eliminated a lot of use of TheadLocal in the implementation on the java.base package.
Hey Dan, why you didn't limit platform thread for virtual thread testing on properties file as you did without virtual thread? So that we could understand that 10 platform thread concurrently handle 60 request without blocking themselves.
Great tutorial ❤
Please make a video how to deploy spring boot app on aws lambda using graalvm
What about
ThreadLocal
and InheritableThreadLocal
is it still going to work wen i switch to virtual Threads
and what about MDC.put from slf4j
It still works with virtual threads, like it did with platform threads
Great! but what is the best practice is i want to enable Virtual Threads for single route in my application ? manually create and run virtual Thread in controller?
You don't enable virtual threads for a single endpoint you do it for your entire application. You don't need to manually create a virtual thread in your controller as Spring is handling that for you.
Nice video. Please what is the name of the software you used to make the slides?
Expected you to show the final console output to make sure not all 10 platform threads were taken. Does someone know what thread names would be printed?
You can run the example and see it in action but you should see the worker thread not go above 1.
I don't think @19:27, there was a need to comment out the server.tomcat.threads.max=10 property before running the example with virtual threads
Actually, keeping the max tomcat threads same would've validated the non-blocking nature of virtual threads. That is what I got when I executed virtual threads example with max tomcat threads as 10. Same result i.e. around 12sec
I have a question regarding how well a component scoped per request with @RequestScope would perform
You are amazing 🎉
Thanks for easy explanation ❤
You just made my day! Thank you for the kind words 🤩
is there a connection pool that is compatible with virtual threads yet ?
Thank you !!!!
great explanation, thanks for share it!
Hey Dan what about home office? What do you think?
What do you mean home office?
@@DanVega i heared that all vmware employees should get back to office. Somehow i cannot link the article.
Does it mean we don't need reactive libraries (like Reactor) anymore?? (If using spring boot)
Great question. If you were only reaching for reactive programming as a scalability mechanism you now have an option on the imperative side of the house. Reactive programming is still a great architecture choice for streams based processing and dealing with back pressure. Another JEP in Project Loom (Structured Concurrency) is still in preview that is something else Webflux solves for.
Ah right, I forgot about back pressure. So @@DanVega from your experience, what are some common problems that can be solved easily using reactive programming and that get complicated when doing imperative? I am thinking about maybe batch jobs, but even spring has Spring Batch right? I don't think it follows a reactive approach.
Hi Dan, was the max thread limit to 10 only to simulate a hardware (cpu core count) bottleneck? bcz I believe even if you didn't enable the virtual threads, the whole request would take similar time since your machine is capable of it even without the virtual threads. Am I correct?
I had that same concern, so I am sure you are right.
I think this depends on the type of the pool type. In case of fixed threadpool here matching with core count, for platform threads was meant to dictate don't create new thread for new request/task, beyond 10. Fixed thread pool has the strategy that till the cap is hit it will spawn new thread, unlike fork join pool. Usually setting more than 10 would pretty much be insignificant. After certain point the performance can vary because of context switches.
Do keep in mind that there is a limit to physical threads you can setup. The default with tomcat is 200 - so while this example did constraint it down to 10 artificially - the same example when scaled would produce comparable results. Virtual Thread can effectively work with no limits (or really high) so they have a lot more scaling than physical threads
Not sure why Dan changed max threads property. I have tested similar code with 300 concurrent threads, when virtual threads are turned on, the app becomes unresponsive. Without virtual threads, it just works fine.
@@htejwani1 that doesn't make sense 😒
Quick question, on my computer I had max 200 threads and it ran. Anyone knows how this is possible? I have 8 cores on my computer.
Thank you!
why this flag is not on by default?
Nice video, it would be cool if you help us understand if a billion request come in at once, there will be a memory issue. Can you force the server to only accept so many active request? I always wondered about that.
There are a few approaches here:
1. Do you want to serve all of those Billion requests? Then you need horizontal scaling, with your app servers sitting behind a load balancer, and preferably having auto-scaling enabled. If there's a DB involved, then you will also need to have multiple read-replicas set up, and a lot of caching between your app and the DB.
2. Do you want to discard requests that you can't serve? In that case, you can set up rate limiting. Any requests that breach your rate limit can be served with an HTTP 429 response (too many requests). They client would have to retry the request once traffic has died down.
Of course, there are many more ways of handling a billion requests, I've just mentioned these two from top of my head.
Hey Dan - Nicely done on the video! Can't wait to try this out myself. Did you look at JVM memory consumption? I'm curious to see what, if any impact on memory there is. Thanks!
Good to hear from you friend 🤩 I haven't done any comparisons with JVM memory consumption but I like where your heads at. I will take a look into that.
@@DanVega - Let me know what you find. I'm curious from a operation cost perspective. ie running ~20 Spring Boot microservices in a K8 cluster. You can chew up a lot of memory quickly, which can drive up costs due to runtime memory needs.
Thank you for the video. I am building the url with query params.If pass the header then not able to pass the query params. If pass the query params then not able to pass the bearer token. How to pass the bearer token in the header?
I got it. Thank you
Hi Dan, can you please tell me how you got the tshirt 😊?
I made a few polos for myself :)
@@DanVega pls tell me ,how I can get same . Please share link if you are fine.
So, in short Virtual Threads is similar to what Kotlin is doing with coroutines, isn't it?
Thanks a lot
You forgot to show the output while running virtual threads!
You can do this on your machine if you want to give it a try.
still light years away from erlang model :)
thanks sir
Your video overlay is hiding a significant portion of your presentation and its annoying
Sorry 🤷♂️