Very good information, but it can much better if you explain how Stream processing works in Kafka , RabbitMQ, Pulsar. Polling and pushing in each techniques is very important to know.
@ByteByteGo - Thanks for another good video. A word of caution. While message queues are impressive, they are often misapplied. I have seen message queues make systems slower and less reliable while consuming a lot of resources that could have been used to speed up other parts of the system. It would be great if you could create a video that explains when (and when not) to use a message queue.
Love your videos. You mentioned exactly once semantics as a feature of Kafka. I have found that another game changer is the capability of building systems that also maintain a guarantee of message order per key (usually a session id) which makes all of it (often in combination with Apache Flink) suitable for a whole new class of applications.
Worth mentioning is also (jetstream) nats, trivial to scale, has routing, partitioning, queueing, pub sub; it also has key value store, object store. Imagine if kafka and redis had a baby.
@@rogsiel While i haven't used their object store yet, i am using the key value store to persist actor's internal state, that way i don't need to rebuild it (event sourcing pattern) before processing the requests. The only issue i have with nats is the lack of backup / recovery options, seems like replication + a recovery cluster is the preferred way.
oh that's cool . Will definitely look into it. Having pub/sub and object store together is interesting. I'm already doing the exact same thing for backups and persistent data.
Nats doesn’t have partitioning, and apparently never will. You can emulate it with channels, but you won’t get rebalancing, for example. So it’s more like a cousin to Kafka
@@nikitamyazin6586 Are you talking about jetstream topics or old nats streaming server (stan)? You can have multiple consumers on the same topic and as long as they belong to the same group they will all act as one. You can also partition topics, based on some (hashed) key and split incoming traffic across multiple workers, so that the same topic always reaches the same worker, which is very useful with actors. I'm sure you've seen this and we're just thinking of different kinds of partitioning. docs DOT nats DOT io/nats-concepts/subject_mapping
Its good know abt latest tech trends and tools but challenging part to learn where to use them in overall system. Breaking a system into different apps is most challenging thing in real world which can only be learn by experience
Hi, I do not know your name, but I really enjoyed your videos and learned so much from them. In the diagram, (0:17min) I think there is a typo in the flow from passenger management to notification shouldn't be "Notify Driver" instead of "Notify Passenger" or invert the flow from the notification queue to passenger. If I'm wrong, please let me know if you have time of course. Hope you have a really nice day or night Mr. and keep going.
The animated flow charts and diagrams are very impressive, which tool did you use to create those? BTW all the videos you created are super high quality in both content and presentation, great job!
What is XXX would be better title. There is no 'vs.' in the whole video. If I just throw this video's content to my tech design spec as explanation of why I picked Kafka over RabbitMQ or another way around, I will probably have to look for another job again. When we talk about 'vs.' we should go from advantages, disadvantages, use case (this was mentioned). No advantage or disadvantage for one over another has been mentioned throughout the video.
This is great information! Thanks for the video. Can you do a video explaining and reviewing some of the top Platforms as a Service options? Eg. Heroku, Render, Digital Ocean, Vercel, Fly, etc. When to use them and which ones are better for which job? Thank you.
Why are you forget Redpanda(and Redpanda ecosystem)?? Redpanda is most powerful streaming solution today working C++ and GO under hood + Kafka Api. Please talk about Redpanda too
I've spent a few years with a RabbitMQ backed system on-prem. Granted, on-prem, on a VM on desktop/laptop, and on-prem in a VM in a llocal data center are all about the same thing these days. Failure modes and recovery aren't always obvious (especially if the VM is frozen and moved), but it's at least a decent solution and might be a great one depending on your circumstances.
It is described as evolution of MQs when it's basically different features of MQs. If one has high load requirements then it's definitely Kafka it is simply an MQ and it's incredibly fast without any feature on top, in it's turn RabbitMQ has many features like shovels, delayed/dead-letter queues almost out of the box. The video is very superficial and definitely does not compare MQs against each other.
Ibm mq supports hardware failures - not so sure. We ran it at a fairy large company - once a new service without proper disposal code took down MQ node, no one could publish or consume a message
From this video, I don't understand how there was an "evolution". It seems you are just naming arbitrary virtues of each messaging system... Very hard to follow. Cheers.
Event-driven microservice architecture is important to know if you want to work in backend. Not sure why you're acting like this is impractical knowledge.
Oh please, you must've beaten all records of unintelligibility in this video. I struggled to understand the simplest words you sais instead of getting the main ideas of the video. Can you please work on your prounciation a little, PLEASE?
Cool, some names of MQ tools were mentioned. Where is the "vs" in "Kafka vs. RabbitMQ vs. Messaging Middleware vs. Pulsar"?
Very good information, but it can much better if you explain how Stream processing works in Kafka , RabbitMQ, Pulsar. Polling and pushing in each techniques is very important to know.
sounds like a good thing for you to work on and post yourself on own channel
@@jayrigger7508 why so arrogant? He literally suggested a potential improvement and you come by and say "Just do it yourself".
That’s not an improvement, but literally a different topic.
@ByteByteGo - Thanks for another good video.
A word of caution. While message queues are impressive, they are often misapplied. I have seen message queues make systems slower and less reliable while consuming a lot of resources that could have been used to speed up other parts of the system.
It would be great if you could create a video that explains when (and when not) to use a message queue.
Do you know anything about that, I'd certainly be useful
wow, within 4min we already get and understand the high level of each message queueing from old to latest. Thank you
Love your videos. You mentioned exactly once semantics as a feature of Kafka. I have found that another game changer is the capability of building systems that also maintain a guarantee of message order per key (usually a session id) which makes all of it (often in combination with Apache Flink) suitable for a whole new class of applications.
Worth mentioning is also (jetstream) nats, trivial to scale, has routing, partitioning, queueing, pub sub; it also has key value store, object store.
Imagine if kafka and redis had a baby.
Wow, didn't know of it. Specially the object store seems cool
@@rogsiel While i haven't used their object store yet, i am using the key value store to persist actor's internal state, that way i don't need to rebuild it (event sourcing pattern) before processing the requests.
The only issue i have with nats is the lack of backup / recovery options, seems like replication + a recovery cluster is the preferred way.
oh that's cool . Will definitely look into it. Having pub/sub and object store together is interesting.
I'm already doing the exact same thing for backups and persistent data.
Nats doesn’t have partitioning, and apparently never will. You can emulate it with channels, but you won’t get rebalancing, for example. So it’s more like a cousin to Kafka
@@nikitamyazin6586 Are you talking about jetstream topics or old nats streaming server (stan)?
You can have multiple consumers on the same topic and as long as they belong to the same group they will all act as one.
You can also partition topics, based on some (hashed) key and split incoming traffic across multiple workers, so that the same topic always reaches the same worker, which is very useful with actors.
I'm sure you've seen this and we're just thinking of different kinds of partitioning.
docs DOT nats DOT io/nats-concepts/subject_mapping
Its good know abt latest tech trends and tools but challenging part to learn where to use them in overall system. Breaking a system into different apps is most challenging thing in real world which can only be learn by experience
Each diagram seems so informative
Thank you
Hi, I do not know your name, but I really enjoyed your videos and learned so much from them. In the diagram, (0:17min) I think there is a typo in the flow from passenger management to notification shouldn't be "Notify Driver" instead of "Notify Passenger" or invert the flow from the notification queue to passenger. If I'm wrong, please let me know if you have time of course. Hope you have a really nice day or night Mr. and keep going.
The animated flow charts and diagrams are very impressive, which tool did you use to create those?
BTW all the videos you created are super high quality in both content and presentation, great job!
What is XXX would be better title. There is no 'vs.' in the whole video. If I just throw this video's content to my tech design spec as explanation of why I picked Kafka over RabbitMQ or another way around, I will probably have to look for another job again.
When we talk about 'vs.' we should go from advantages, disadvantages, use case (this was mentioned). No advantage or disadvantage for one over another has been mentioned throughout the video.
cry nub
Thank you for continuing to create such helpful resources! 🐐
Should the arrow from driver management to notification be labeled as "NOTIFY DRIVER"?
Which tool has been used for this graph can anyone tell me
This is great information! Thanks for the video. Can you do a video explaining and reviewing some of the top Platforms as a Service options? Eg. Heroku, Render, Digital Ocean, Vercel, Fly, etc. When to use them and which ones are better for which job? Thank you.
what service do you use to draw these animations?
Why are you forget Redpanda(and Redpanda ecosystem)?? Redpanda is most powerful streaming solution today working C++ and GO under hood + Kafka Api. Please talk about Redpanda too
No 😂
vendor lock in ecosystems get outta here
If we buy ByteByteGo subscription will you share your powerpoint clip arts ?
I don't like the cloud focuses solutions. Like the latest you showed. I want to run a message queue system on prem.
I've spent a few years with a RabbitMQ backed system on-prem. Granted, on-prem, on a VM on desktop/laptop, and on-prem in a VM in a llocal data center are all about the same thing these days. Failure modes and recovery aren't always obvious (especially if the VM is frozen and moved), but it's at least a decent solution and might be a great one depending on your circumstances.
While Pulsar is evolution of a message queue in one way, it lacks exactly once delivery policy, which is Kafka is famous for the most
It is described as evolution of MQs when it's basically different features of MQs. If one has high load requirements then it's definitely Kafka it is simply an MQ and it's incredibly fast without any feature on top, in it's turn RabbitMQ has many features like shovels, delayed/dead-letter queues almost out of the box. The video is very superficial and definitely does not compare MQs against each other.
Don't forget about Microsoft Message Queue (MSMQ), it's still there.
how about NATS?
Great video
How do you design animated diagram?
can i ask what tool to created animated diagrams like that ?
How do you make these slides? What is the tool name?
I looked into Pulsar recently, looks really powerful. Anybody got real-world experience already?
Thank you for doing this!
Excellent! Many thanks :-)
How do you create the animations? After Effects or built-in animation with Power Point?
anyone have used ZeroMQ?
How do you exclude NATS from here ? Come on man !
Ibm mq supports hardware failures - not so sure.
We ran it at a fairy large company - once a new service without proper disposal code took down MQ node, no one could publish or consume a message
hin and goog luk, please what s the sofwre wich you make your presentation
how about rocketmq？
I see Kafka is often used as message queue which it is not.
Persistent sharded logs as the source of truth 🥵
please do Spark and Hadoop
theres an incredibly nasty high pitch noise until 0:12
We would need implementation too for all the tools you have explained.
Please make videos on those too.
How about Redis Pub/Sub?
From this video, I don't understand how there was an "evolution". It seems you are just naming arbitrary virtues of each messaging system... Very hard to follow. Cheers.
Which you had included product from Microsoft in this comparison.
Programming content creators: Want to know how companies you’ll never work for do things the companies you will work for don’t need to do?
Or giving you freshest framework/tool so you can rewrite your application again for no reason at all 😅
@@oskarszwajkowski Trend casing, when you need an excuse to not do real work
U are just mad because u cant handle more advance concepts than programming.
@@matonolo "More advanced concepts than programming".. what?
Event-driven microservice architecture is important to know if you want to work in backend. Not sure why you're acting like this is impractical knowledge.
how about rocketMQ?
I use only Kafka. May be I nee to try something new.
RabbitMQ is fine 😊
one day i will understand all this lol
Jaunita Ridge
Cordelia Fields
Collins Meadow
Bro i think you are well knowledgeable but to help,why not indicate by doing than just explaining?I have never seen you code or apply these concepts.
Treutel Island
🫡
Oh please, you must've beaten all records of unintelligibility in this video. I struggled to understand the simplest words you sais instead of getting the main ideas of the video. Can you please work on your prounciation a little, PLEASE?
There's a transcript option
this is a you problem, video is well explained. maybe you should learn some basics first.
Kafka uses an inefficient pull mechanism where consumers keep polling the broker. With time it will lose steam like Cassandra