► What should I test next? ► AWS is expensive - Infra Support Fund: buymeacoffee.com/antonputra ► Benchmarks: th-cam.com/play/PLiMWaCMwGJXmcDLvMQeORJ-j_jayKaLVn.html&si=p-UOaVM_6_SFx52H
Id like to see a benchmark with the differences between microservices comunications : grpc, rest and rabbitmq -latency , memory and cpu using any linguage is fine
Based on your tests all of these frameworks performed really impressively. Honestly the differences seem to be rather small and I feel like using any of them would give you super high performance web server.
its amazing how the differentials are so minute the availability graph scales to show difference, but in great scheme of things they're all still within 99% available: thats crazy rust is so performant 😯
Please understand that raw req/s is not a metric to select a framework. Its a lot of different things. Top priority should be ergonomics and ease of implementation. The moment your app needs to handle 10k/s it's a huge scale already
I love your videos, you are the channel I've seen that makes the best benchmarks, putting them to the test as it would be in the real world and with a clear way to see the results, keep it up :) 🎉
Correct me if im wrong, but actix utilizes a more singular request to thread strategy while axum uses a more work pool threading strategy. Considering they are
@viniciusataidedealbuquerqu2837 stateless requests are more async/thread friendly but then someone needs to handle the state in the interim. I was attempting to reference the nature of how load is assigned to threads. The result is that axum FEELS nicer to build more complex middleware and such, while actix feels much more straightforward. But that's just my personal experience.
@@theintjengineer I just did this test yesterday and yes Bun+Elysia is super performant. I did Bun vs Elysia and Bun got 47k rps while Elysia 60k rps in a 1 thread 10 connections setting. Any idea what's the optimization done by Elysia to achieve such performance?
Fantastic! It seem like there is not much performance difference between them. If I ever need to write a web application in Rust, I think I will go with the one that is easier to use and has better community support.
Do you have an idea what causes the degradation of performance? The usual suspects mem ans cpu seem to be good… Could you maybe have reached the host socket limits?
i don't think so, but i'll double check next time. It's a eks cluster and usually you don't directly manage nodes. i may try to using serverless eks (fargate) next time
The combo of all services failing at the same time + no service being near their resource limits + no service p99 latency being near what would be required to cause a 408 response would indicate that some other limit had been reached before the services themselves degraded.
Was looking for this. This test is testing the aws nodes and not the frameworks. Great test setup otherwise and hope you upload a rerun after figuring out what limit you hit.
This is actually a pretty good point, iirc the default limit in linux is somewhere in the 25k-32k range. And if they're all pushing the limits of the OS, that's insanely impressive for all of them.
Very interesting charts. But what’s the takeaway? All three seem to perform similarly. Is it fair to say that the choice depends on developer experience? Or is there something else to consider at runtime?
So far, yes, it's about developer experience. I'll keep adding more metrics and graphs, and over time, I may discover additional behavioral differences.
26k seems pretty close to ephemeral port exhaustion. You may want to look into either increasing this limit or allowing the load generator to re-use connections between requests (Connection:keep-alive) or tweaking the load generator to not have more than that many connections open at a time.
This _potentially_ looks like it might be hitting a different bottleneck. Maybe like the ingress controller or load balancer or something. I know from load testing my own Rust web application on K8s, the thing we max out first is the load balancers (haproxy in my case) rather than the app itself. Anyway, it looks very similar to what we're observing here - the app never gets close to 100%, but the load balancer pods are maxed out. A good way to rule this out is to run the app directly on an EC2 machine and hit it with wrk, or something equivalent that can generate enough load without hitting limits itself. Then we'll know how fast it actually is before plugging it into K8s. You might have checked this already, of course - but would be interesting to know :)
Thanks! I don't use a load balancer or ingress controllers in this video, only native Kubernetes networking (Service of type ClusterIP). yaml - github.com/antonputra/tutorials/tree/main/lessons/206/deploy/actix-app
I had exactly the same reaction. All three frameworks are failing at about the same time while all the other metrics seem fine. Why would it stop serving if the latency is way below 1 mSec and the CPU (40% ish) and memory loads (1.5% ish) are fine? What else could be causing the failure
@AntonPutra I don't think it would resolve the domain name that often Did you check other constraints such as networking or container limitations? Could you run the test on bare metal or in a VM?
Hi Anton， love your content! I’m recently learning Prometheus and want to know your promql for generating those benchmarks diagrams! Can you share those with me? Thanks!
One thing that might be interesting would be to flip this around. Pick one of the implementations and try to compare similarly sized/priced options between aws, gcp and azure.
For the slower (Python) frameworks, there are methods to make them more performant rather easily. 1. Use FastWSGI for Flask or other sync based frameworks. 2. Use Granian for Quart/FastAPI or other async based frameworks. Yeah they are slow, but to make them moderately fast is easy. Note they're never gonna reach the same levels as Go/Rust/Zig.
Did you test different async runtimes? For latency maybe a thread per core framework will work better like bytedances Monoio or a non-async frame work.
The fact the three start failing at the same moment, tells there is something common between the frameworks that doesn't tolerate the load, or perhaps the problem is outside.
Hello, a real question. If there's CPU resource (max 40%) and RAM available, why does the application start failing responses? Where's the bottleneck, what's missing? Disk? Connections?
I couldn't find the source for "aputra/utils-tester-amd64" but if you exceed 128 concurrent connections the Rust servers will stop processing new connections by default. I'm not sure why the Rust stdlib has such a low default for max concurrent connections - but that might be why all of the servers all tanked at around the same req/s despite having plenty of CPU and RAM to work with. I raised a PR for your previous test increasing the max connection count
For sure. I personally enjoy Rust because of the type safety and the language features. I like Axum because it seems well designed. The performance is a pleasant side effect and also makes it cheap to host the stuff I create.
0.7, i provide a link to the source code under each of my videos - github.com/antonputra/tutorials/blob/main/lessons/206/axum-app/Cargo.toml#L7C9-L7C12
Even tho I believe Leptos has SSR capabilities, Leptos is more focused on the client than the server. It is a frontend framework that works with web assembly to run on the browser.
I though Rocket would be even worse. And I though Axum would be better. I am still a big fan of axum, but there seams to be less development on it recently. I really thought axum would clearly win the rust web war over time, but I am less certain now.
Wow, that seems like a lot of unnecessary over engineering, to setup a multi-node k8s cluster on a greatly-multi-core nodes, and to limit rusts notoriously multithreaded async runtimes to two cores. I get that it is more closely emulating a production-like environment, but I dunno. Seems like this additional complexity doesn’t buy you much.
@@AntonPutra the big question for me is the inflection point where value switches from scaling lambdas to servers running fargate ECS and ec2 reserved instances. The numbers are of course heavily determined on what you want them to do, but it'd be nice to see rules of thumb, eg: 15kreq/sec for random crud updates is when ECS is more cost effective than lambda, so if you're under that stay serverless
Looks like you're using the multithreaded flavor runtime for Tokio, but for containers I would choose the single thread flavor, since you don't have CPU cores, and the workstealing overhead is quite substantial.
I'd argue it isn't overkill if it means you can reduce your hosting costs by a large margin. To get close to this in ruby/python you'd need many more instances running, load balancers, k8s (maybe). Where here you can run one instance and not need to bother with any of that complexity
i'm curious how well rust can replace data pipelines that so many companies have, for example storm topos, spark/flink and just in general kafka consumers and producers...
@@AntonPutra feels like a good use case for processing/streaming large amounts of data. Calculate more with the same infrastructure and get better insights? I'd guess the largest blocker for most companies is just the ecosystem vs doing something custom
► What should I test next?
► AWS is expensive - Infra Support Fund: buymeacoffee.com/antonputra
► Benchmarks: th-cam.com/play/PLiMWaCMwGJXmcDLvMQeORJ-j_jayKaLVn.html&si=p-UOaVM_6_SFx52H
Axum vs Fibre
web socket connections test in go/rust
Elixir phoenix vs rust axum
Laravel (PHP) vs Fiber (Go) vs Axum (Rust) vs Django (Python)
Id like to see a benchmark with the differences between microservices comunications : grpc, rest and rabbitmq -latency , memory and cpu using any linguage is fine
Based on your tests all of these frameworks performed really impressively. Honestly the differences seem to be rather small and I feel like using any of them would give you super high performance web server.
true
I guess victory goes to just which one is easier to work with for quick results
@@nsing323 true
its amazing how the differentials are so minute
the availability graph scales to show difference, but in great scheme of things they're all still within 99% available: thats crazy
rust is so performant 😯
💯
Great content, I was actually thinking of whether I use Axum or Actix in my project though my first choice is Actix. Now Im decided to stick with it.
Please understand that raw req/s is not a metric to select a framework. Its a lot of different things. Top priority should be ergonomics and ease of implementation. The moment your app needs to handle 10k/s it's a huge scale already
well actix has more starts 😊
I see a video from Anton, I click and watch it. It's that easy ;-D
haha thanks!
I love your videos, you are the channel I've seen that makes the best benchmarks, putting them to the test as it would be in the real world and with a clear way to see the results, keep it up :) 🎉
thank you! i'll keep adding more metrics in the future just for the reference
Your videos are the best! Keep going, I saw a comment Zig vs Rust, this will be very interesting!
yes it is coming next
@@AntonPutra Thank you very much!
Damn that's a lot of crabs 🦀🦀🦀
🤣
Here you are again
Correct me if im wrong, but actix utilizes a more singular request to thread strategy while axum uses a more work pool threading strategy. Considering they are
stateless tends to be more asynchronous no?
@viniciusataidedealbuquerqu2837 stateless requests are more async/thread friendly but then someone needs to handle the state in the interim. I was attempting to reference the nature of how load is assigned to threads. The result is that axum FEELS nicer to build more complex middleware and such, while actix feels much more straightforward. But that's just my personal experience.
@@shm236 gotcha thank you
interesting
You should try Nodejs vs Bun vs Deno
Also express fastify Elysia hono
will do!
I would like to see a clustered fastify web server vs go std lib
Nodejs and Bun+Hono lose against Bun+Elysia. Elysia is really fast.
@@theintjengineer I just did this test yesterday and yes Bun+Elysia is super performant.
I did Bun vs Elysia and Bun got 47k rps while Elysia 60k rps in a 1 thread 10 connections setting.
Any idea what's the optimization done by Elysia to achieve such performance?
boah many many thanks this was the most helpful benchmark for me :)
my pleasure :)
Fantastic! It seem like there is not much performance difference between them.
If I ever need to write a web application in Rust, I think I will go with the one that is easier to use and has better community support.
axum looks cleaner
Awesome video, thanks! It'd be interesting to include salvo too, since it claims to be the fastest Rust web framework!
interesting, ok!
Do you have an idea what causes the degradation of performance? The usual suspects mem ans cpu seem to be good… Could you maybe have reached the host socket limits?
i don't think so, but i'll double check next time. It's a eks cluster and usually you don't directly manage nodes. i may try to using serverless eks (fargate) next time
~25000 limit is very suspiciously associted to some hardcoded limit
The combo of all services failing at the same time + no service being near their resource limits + no service p99 latency being near what would be required to cause a 408 response would indicate that some other limit had been reached before the services themselves degraded.
Was looking for this. This test is testing the aws nodes and not the frameworks. Great test setup otherwise and hope you upload a rerun after figuring out what limit you hit.
This is actually a pretty good point, iirc the default limit in linux is somewhere in the 25k-32k range. And if they're all pushing the limits of the OS, that's insanely impressive for all of them.
Very interesting charts. But what’s the takeaway? All three seem to perform similarly. Is it fair to say that the choice depends on developer experience? Or is there something else to consider at runtime?
So far, yes, it's about developer experience. I'll keep adding more metrics and graphs, and over time, I may discover additional behavioral differences.
Great video thanks for sharing. Can you please compare Rust Actix and Go Fiber ? If already done - can you please share the link? Thank you so much 🙏
i have one 2 videos ago, but i didn't push it to the end.. i'll update soon
@@AntonPutra Awesome 😎 Thanks allot 👍
26k seems pretty close to ephemeral port exhaustion. You may want to look into either increasing this limit or allowing the load generator to re-use connections between requests (Connection:keep-alive) or tweaking the load generator to not have more than that many connections open at a time.
thank you! i've improved those benchmarks in the latest videos
This _potentially_ looks like it might be hitting a different bottleneck. Maybe like the ingress controller or load balancer or something.
I know from load testing my own Rust web application on K8s, the thing we max out first is the load balancers (haproxy in my case) rather than the app itself. Anyway, it looks very similar to what we're observing here - the app never gets close to 100%, but the load balancer pods are maxed out.
A good way to rule this out is to run the app directly on an EC2 machine and hit it with wrk, or something equivalent that can generate enough load without hitting limits itself. Then we'll know how fast it actually is before plugging it into K8s.
You might have checked this already, of course - but would be interesting to know :)
Thanks! I don't use a load balancer or ingress controllers in this video, only native Kubernetes networking (Service of type ClusterIP).
yaml - github.com/antonputra/tutorials/tree/main/lessons/206/deploy/actix-app
I had exactly the same reaction. All three frameworks are failing at about the same time while all the other metrics seem fine. Why would it stop serving if the latency is way below 1 mSec and the CPU (40% ish) and memory loads (1.5% ish) are fine?
What else could be causing the failure
@@CalifornianViking it could be dns, cause i was using service name
@AntonPutra I don't think it would resolve the domain name that often
Did you check other constraints such as networking or container limitations?
Could you run the test on bare metal or in a VM?
I agree with this. TechEmpower’s Actix gets over 1M RPS, so failing at 26k is a red flag.
A failure mode where latency goes up to 2 ms and "only" 99.8% availability is fun
well it was done on the same vm without any external dependencies so i think it is reasonable
Hi Anton，
love your content! I’m recently learning Prometheus and want to know your promql for generating those benchmarks diagrams! Can you share those with me?
Thanks!
sure - github.com/antonputra/tutorials/tree/main/lessons/135
Wow! Thank you! Axum forever :)
Actix forever :)
fiber beats all of them 😂
One thing that might be interesting would be to flip this around. Pick one of the implementations and try to compare similarly sized/priced options between aws, gcp and azure.
thanks, i was thinking about it. will do in near future
Please try Fastapi vs Robyn vs Flask next time.
ok noted!
All equally slow lol
For the slower (Python) frameworks, there are methods to make them more performant rather easily.
1. Use FastWSGI for Flask or other sync based frameworks.
2. Use Granian for Quart/FastAPI or other async based frameworks.
Yeah they are slow, but to make them moderately fast is easy. Note they're never gonna reach the same levels as Go/Rust/Zig.
I use axum for api gateway, hope it perform well when deploy on a vps
it's very close to actix..
Great video! Just FYI the LTO optimization gets disabled when you set codegen to 1.
interesting, is it in the docs or i just missed it?
@@AntonPutra Actually, my bad. If you set LTO to false and set codegen to 1, it won't perform any LTO optimization. I must have read it wrong.
@@zacksalah5879 no worries
Did you test different async runtimes?
For latency maybe a thread per core framework will work better like bytedances Monoio or a non-async frame work.
i may test it in the future, but in container it is tricky
The fact the three start failing at the same moment, tells there is something common between the frameworks that doesn't tolerate the load, or perhaps the problem is outside.
yes, it's infra issue, more precisely the issue with a go client :)
ty for this video
🫡
It would be nice to compare the performance of Hyper with the simplest implementation based solely on Tokio.
noted!
Can you do rust(actix) vs nodejs(express) so I can convince my team and manager to use rust over nodejs 😅
haha, yes soon
Same here.
What benchmark graphing tool is he using? Never mind -- grafana and prometheus . I had to watch the video again. I missed it the first time around.
Please make a video on "Deno" vs "Bun" vs "Node 22"
ok, noted!
Nice stuff
thank you!
Can u do video how to create this comparison pls
sure at some point
Hello, a real question. If there's CPU resource (max 40%) and RAM available, why does the application start failing responses? Where's the bottleneck, what's missing? Disk? Connections?
possible socket count but i haven't tested this yet, take a look at this PR - github.com/antonputra/tutorials/pull/264
Can we also include Node just to see the difference between Rust and JS performance?
yes, soon
🔥
❤️
I like the developer experience with Axum. It's good to know that it's not that bad compared to Actix.
it is very close
I couldn't find the source for "aputra/utils-tester-amd64" but if you exceed 128 concurrent connections the Rust servers will stop processing new connections by default. I'm not sure why the Rust stdlib has such a low default for max concurrent connections - but that might be why all of the servers all tanked at around the same req/s despite having plenty of CPU and RAM to work with. I raised a PR for your previous test increasing the max connection count
thanks! merged i'll try in new video
Rocket vs Jetzig should be interesting. Both in Release mode please 😅
ok noted :)
How many concurrent users did you tested and is it feasible to test 1000 users as
ab -n 2000 -c 1000 -k localhost/test
20 pods starting with 1 cient ~> 160 so max was 20*160 = 3200
26K req/s is plenty enough for me :)
For almost all the normal cases, it is overkill.
For sure. I personally enjoy Rust because of the type safety and the language features. I like Axum because it seems well designed. The performance is a pleasant side effect and also makes it cheap to host the stuff I create.
💯
It would be interesting to see examples using python (fastapi or django) or js
ok, noted!
What version of Axum? 0.6 or 0.7?
0.7, i provide a link to the source code under each of my videos - github.com/antonputra/tutorials/blob/main/lessons/206/axum-app/Cargo.toml#L7C9-L7C12
Which framework did you like to write code the most?
not rocket :)
Maybe go vs php? Because a lot of companoes rewrite their php projects on go. So, go is a php replacer for them.
sure i was thinking about php for some time
Why does rust start falling over even thou it does not max out CPU or ram? Is this more the networking interface falling over?
that was an infra issue, i've made some improvements after that
would appreciate something like bun vs actix
thanks! noted!
I'm new to the Rust ecosystem. Shouldn't Leptos be also included in this test?
I thought Leptos is the most popular web framework for Rust... 🤔
Even tho I believe Leptos has SSR capabilities, Leptos is more focused on the client than the server. It is a frontend framework that works with web assembly to run on the browser.
@@galower405 Oh, I see, thanks for the explanation.
webassembly is coming soon
I hope your EC2 instances had dedicated CPU cores so there was no problem with shared resources
I usually run these tests at least three times before recording, so the results are always consistent, and I use new EC2 instances each time.
I though Rocket would be even worse.
And I though Axum would be better. I am still a big fan of axum, but there seams to be less development on it recently.
I really thought axum would clearly win the rust web war over time, but I am less certain now.
the difference is very small between axum and actix
how about node vs bun vs deno?
yes coming soon
Please do Java Spring Boot (Native) vs Spring Boot (JDK) VS Quarkus (Native) vs Quarkus (JDK)
thanks will do
pls compare actix against web servers made for non-usual async runtimes like hyper using smol runtime and submillisecond (lunatic runtime)
ok noted
Next C++ vs Rust vs Zig based web frameworks please....
zap (zig) is comming next
Why does axum have a better score on techempower benchmarks?
I’m not sure, but I’ll do some more testing soon, covering the top Rust frameworks
Is it possible to get something 100k rps with 100ms latency?
yes i got 150k rps with 2 cpu, in a new video
@@AntonPutrawhen are you posting mate and also with how many concurrent users did
Wow, that seems like a lot of unnecessary over engineering, to setup a multi-node k8s cluster on a greatly-multi-core nodes, and to limit rusts notoriously multithreaded async runtimes to two cores.
I get that it is more closely emulating a production-like environment, but I dunno. Seems like this additional complexity doesn’t buy you much.
Well, that's a typical deployment. I may go over each optimization technique I've got so far in the future.
Rust is the best!
one of the best for sure
Nodejs vs Rust please
ok noted!
+1 please mate
Garcia Shirley Hernandez Edward Wilson Jason
?
Go vs Elixir please
ok noted!
Some love to python
soon
I'd like to see using Rust w/ lambda w/o a web framework vs using the framework at all.
i have older video comparing lambdas but i'll definitely make a new one
@@AntonPutra the big question for me is the inflection point where value switches from scaling lambdas to servers running fargate ECS and ec2 reserved instances.
The numbers are of course heavily determined on what you want them to do, but it'd be nice to see rules of thumb, eg: 15kreq/sec for random crud updates is when ECS is more cost effective than lambda, so if you're under that stay serverless
@@arimill1045 ok got it
do haskell vs rust
noted
Looks like you're using the multithreaded flavor runtime for Tokio, but for containers I would choose the single thread flavor, since you don't have CPU cores, and the workstealing overhead is quite substantial.
thanks for the tip
Moore Kenneth Lopez Shirley Williams Angela
?
It's all not different.....0.5 or 1 percent is not significant in real environment.
laughs in Javascript 💀
😊
Sir, C++ vs Rust.
thanks, noted!
Just goes to show how overkill rust is for server apps
I'd argue it isn't overkill if it means you can reduce your hosting costs by a large margin. To get close to this in ruby/python you'd need many more instances running, load balancers, k8s (maybe). Where here you can run one instance and not need to bother with any of that complexity
i'm curious how well rust can replace data pipelines that so many companies have, for example storm topos, spark/flink and just in general kafka consumers and producers...
@@AntonPutra feels like a good use case for processing/streaming large amounts of data. Calculate more with the same infrastructure and get better insights? I'd guess the largest blocker for most companies is just the ecosystem vs doing something custom