What is a Message Queue and When should you use Messaging Queue Systems Like RabbitMQ and Kafka
ฝัง
- เผยแพร่เมื่อ 1 พ.ค. 2020
- Message Queues system like RabbitMQ and Kafka are amazing technologies but when should you actually use a message queue? I discuss this in this video.
Kafka Course • Apache Kafka Crash Course
RabbitMQ Course • RabbitMQ Crash Course
Pub/Sub Video • Publish-Subscribe Arch...
🏭 Software Architecture Videos
• Software Architecture
💾 Database Engineering Videos
• Database Engineering
🛰 Network Engineering Videos
• Network Engineering
🏰 Load Balancing and Proxies Videos
• Proxies
🐘 Postgres Videos
• PostgresSQL
🚢Docker
• Docker
🧮 Programming Pattern Videos
• Programming Patterns
🛡 Web Security Videos
• Web Security
🦠 HTTP Videos
• HTTP
🐍 Python Videos
• Python by Example
🔆 Javascript Videos
• Javascript by Example
👾Discord Server / discord
Support me on PayPal
bit.ly/33ENps4
Become a Patreon
/ hnasr
Stay Awesome,
Hussein - วิทยาศาสตร์และเทคโนโลยี
2020 is not all bad..... we learn a lot from your videos. Thanks for being there for us.
Extend this comment to 2021. I have tons of content at my disposal from my company but still come back to youtube for some basic stuff. I love this!
This comment aged bad. All IT professionals are in trouble atm
What really exemplary about your videos is that you first explain the need and then explain the tools, keeping "necessity is the mother of invention" intact.
You have a wonderful passion for Software Engineering. You just pulled me into this awesome ocean of software engineering. I'm super addicted to you channel. I've been watching all you videos continuously. Thank You🙏
Ravilochan thank you for beautiful words 😊
@@hnasr i don't know why but it feels like you made your videos just for me, that you are explaining only to me....what i mean by that is that your videos ware not made just for the sake of covering a topic and upload to youtube....you really care a lot to make sure your viewers understand the concept you are talking about...very personal touch in your videos...as a self-taught frontend developer, videos like this help me understand the backend stuffs in a more frienddly way and feel less like a imposter...you are awesome and keep doing this hussain :) thank you very much !!
I am overwhelmed by the amount of the number of videos you post. I just found out your channel and it seems everything you post is essential for learning. So from today, I have decided to just watch any video you post as soon as possible to keep up with you.
The way you explained this was extremely clear. I just landed a new job as a software engineer and the technology this company uses is Kafka. I took a messaging queue as something literal to actual messages. This makes a lot more sense now.
Love your teaching style Hussein. I enjoy you always making it a fun experience -- it allows me to consume more educational content without feeling exhausted. I also really appreciate your commentary on opinions and habits of engineers around each video's topic, and your comparisons between different technologies -- it adds more context and useful information that you can't find in other educational content. Thank you for making these videos, and best wishes to you!
Skylar Kerzner thank you Skylar! I am so happy you are enjoying the content dear. 😊
Best TH-cam Channel for almost every concepts in backend
I was wading through "JMS" search results on TH-cam and stumbled upon your video. And to my luck I found (a gem) somebody who explains concepts with such an enthusiasm.
This is the first time I am subscribing to a channel just by watching it's first video itself.
Not exactly what I was looking for, but the explanation was so clean and clear that I stuck around for the whole thing! Great video!
One of the most brilliantly simple explanations on TH-cam. Thanks a lot, Hussein!
❤️
Really appreciate how you explain complex topics in an easy way.
Nice! This method of consuming knowledge through a "virtual talk" is the next level. Entertainment and experience in one box!
I always feel like first liking the video and then start watching it. This channel is the greatest i have seen so far
With the power of this one video, you have become one of my favourite people on the internet.
one more video consumed! awesome!
Excellent video, We need more of this kind of videos help to decision making, identify and understanding the problem then analyze what tech/method can solve or minimize the problem.
also finally understood the difference of queue and pub/sub 10:52 .
thanks for share your knowledge.👍
thanks hussein for the video! i didnt understand message queues at first, but you literally covered all my questions in this video.
That answered lots of questions I had, and some I didnt have but should have had as well. Well done and thank you, subbed!
When you said “think whether you need a queue”, I actually did, and realized I don’t! I thought of a much simpler solution, thanks to that intro, LOL :D
I recently implemented a simple queue system in mysql and php. I store all the jobs in a table with all the data I need. And then for each kind of job I have written python script to perform the job.
It sends emails to the users, developed api's for dashboard where user can see the results,state etc and now me watching this video makes me confident about my solution :)
This is cool!!! your design is very simple and elegant for your use case
Interesting, but sending an email doesn't look like very heavy task, why do want to add extra latency in db read and writes? and also, have you compared the performance to a simple processing?
@@navjot7397 actually here job can be anything from pdf files generation to web crawling etc iam sending email notifications regarding job status an built a dashboard on top my db to show the status of the job.
@@azr_sd got it. cool mate
My All-time favorite channel !!
Thank you for this explanation. Took a few videos to find this one! Great examples. Figuring all this stuff out when new to this field has definitely been a steep climb. Definitely needed this.
What a lovely way to explain Hussein , right to the basics and answering the fundamental question , WHY ??
I love your explanation. I’ve been moving to backend development and these things or issues I am seeing a lot
Thank you 👍🏽 this clears conceptual and architecture views
My pleasure Arun!
what an amazing explanation thank you very much
At the end you mentioned that the web server
should not process your request, it should just
respond. My question is, what are the other options
a developer should use to process request/data?
Should it be another server/service that the web
server then makes a API request to? Or maybe the
web server then connects to a message broker and
sends the data as a job to a server/service that just
handles processing the data for that specific job? I
know this video is from 2 years ago but the
information is timeless, thank you for all the content
you've created for the community!
Whoa, I'm reading the same book on your shelf- Made To Stick!
If you wanna do a "instant" response with a standard REST, well, return first and do the heavy job after.
Amazing video Hussein! Thanks for sharing!
Nice explanation about queue. Awesome.
Thank you!
5 mins in the video , I immediately subscribed
You are amazing! The explanation is so clear! :D
I really respect you and appreciate your valuable insights. Thank you very much for sharing your knowledge.
well explained!
If there were some visual resources, I would understand more easily but this also helped me a lot!
Good explanation
I started to love the system design stuff.
Great explanation man
I love how you explain concepts.. new fan!
This is really great
Please can you make a roadmap based on your channel for beginners, there's really a lot to learn from here but the paradox of choice is really big
Awesome content and explanation as always, thanks Hussein
Hello @Hussein Nasser.
I realy love your channel, i don't know how i never saw it before, thanks a lot for your great content !
Just one question, in the case you are talking we can still use just a normal Database to save the jobs that we need to run, it still dosen't explain why the need of a message queue on top of database.
amazing tutorials thanks Hussein
Cool... U explained it
I can't tell you how many times I've circled back to Big O notation and it still escapes me - 7:32 Great video.. thanks for sharing
awesome!! One question tho, I want to build a platform for a nationwide grocery store, queuing system is a good choice, right?
Thank you for an awesome explanation. But yes as a norm after a perfect explanation and understanding questions arise. So my question is if we use RabbitMQ / Kafka for long processes it will use the same server resources right ? Then why not we use async requests to do so ? I mean it will work in same way right that it won't halt your second request but still be working on first request on same time? Another question is why we use RabbitMQ / Kafka but not Hangfire ? Please help to understand the differences with a simple example. May be it will be helpful for others. Thanks
Thank you so much
Thank you for the explanation. You should use a graphics tablet. It is well understood when you draw while explaining.
Excellent
Man you just awesome.. I wish you had a full fledged backend course..
Thanks Ruhan. I don’t even think it’s possible to make such course. Backend engineering is a continuous learning process and can’t be learned in an hour or two. Check out my playlist th-cam.com/play/PLQnljOFTspQUNnO4p00ua_C5mKTfldiYT.html
Thank you so much ❤
Man, you are amazing, I can't thank you enough for posting these videos and making technologies such as RabbitMQ and Kafka that were giving me and incredible hard time so approacheable!
Can you guys help me?
I'm creating a dashboard.
So, the goal is to work with stats to provide the user an overview of how the things are going for the past months and the other part is to fetch data from the social media to see how the public is reacting to it. Is this a good example to use a Messaging Queue?
I tried to implement Kafka on windows, using NodeJS. I was able to create a topic, send some simple messages and consume them.
(I used kafka-node)
My view (and doubt) about this is: I have my PostgreSQL DB and my node backend API. Will Kafka run in beetwen them?
I mean, if I want to make a request, first I will subscribe to a certain kafka topic, which will have the ability to connect with the DB and get the information I want. Is that how it works?
I'm having trouble connecting Kafka to my DB using nodeJS as well.
I hope I was as clear as possible. Sorry about my english.
Thanks in advance.
Greetings from Portugal!
Please make a video on eventsourcing, how they are related or different from message queues. I am confused with it. TIH
That would be great.
Assalamu 3alaykom Hussein, good content, i have a question tho. Whats the difference between the asynchronous nature of something like nodejs and a messaging queue system like RabbitMQ?
Could you please make a video on kafka? Explaining about brokers and how fast can we process data and performance perspective?
ram varma its your lucky day 😊 Apache Kafka Crash Course
th-cam.com/video/R873BlNVUB4/w-d-xo.html
little people nowadays can illustrate these topics in laymen terms, keep going
Thanks 🙏
thank you !
Hi Hussein. Great Video as always. Just a side question - How do you make these beautiful thumbnails? Any special software?
Thanks Vaibhav! Ye I use Canva , I am really bad at photoshop
amazing!
You are awesome, though you only know about technology and not about algorithms, but hey, your videos are fabulous, totally became a fan!
Thank you 😊 correct I have many limitations one of them are algorithms another thing is frontend dev. Glad you are enjoying the content ❤️
But knowing technologies is a great thing too, the one I lack, that's why your videos are so awesome, the one in which you've explained the OSI model is fantastic, what the actual fuck you talked about, no one literally never taught it so elegantly. Thanks! M just an undergrad hahahah!
nice one
Excellent. Use diagrams too. 🙏🙏
Thank you
🙏
Actuallty NodeJs has more than one thread... because the event loop uses libuv and kernel primitives for the event loop. Resulting in more than one thread... it can take all the multithreads of your machine for async.
I’ve started to press like button before watching
that's amaaaizing 😂😂😂
Can you also talk about message and task queues?
Can queue can also help with being losseless? If server fails handling the request he lost it but if it was in some queue maybe they can help how?
Whats the difference/benefit over something like socket io ?
Please, make a video on the newest RabbitMq feature Streams.
So should I learn it just for the sake of knowing it, or just wait until its a necessity in a project ?
Can any one help me to guide what is the best solution for bellow app?
I want to create a pos system that can sync data on local network.
i.e. their are 3 pos system on store and all connect on same network. If any pos place any order, that data should be sync to rest 2 pos. And ya also any large data using full sync.
Should I use ZeroMQ for syncing locally?
what could be better when creating a chat service (audio and video) =>Web-sockets OR RabbitMQ
Both
A queue would also help you if the ordering of requests really matters to your system. HTTP communication won't guarantee this because things are asynchronous
Thanks so much for this video tutorial.
Hi Hussein,
Thanks for the tutorial. One thing that is confusing me, is MQ is just another layer that could be fit in front of the existing API or we have to implement MQ at the very beginning stage of the development?
What i meant is, suppose I have developed an api that works using request, response model and after some days due to heavy usage and heavy load application can't serve consistently so I want to implement MQ on the top of existing application. Is this sounding a good?
And also please clarify, suppose My api have two endpoints, one- send email, two-receive email. So, i can create two copy of my application and in RabbitMq i can create two queues, for send email queue, i can bind it with copy1's send email method and for received email queue i can bind it to copy-2's receive email method. Is this how it should be done?
Moving from request/response to a queue is a big task. I think you should anticipate that load and design your application in that way. Alternatively you can have sync vs async endpoints and route your clients accordingly
@@hnasr Thanking again for your prompt response. That clarifies all of my confusions.
If your process is indeterministic, long running or resource hungry ... Thanks Nasser
thank you very much for talking about queues. can you give an openion about sleeping workers in a queue. in my case, it is not cpu bound. i have to call an api periodically to check for a specific transaction status. so i didn't go for rabbit mq or any other packages.
i used gevent greenlets as workers. will it cause any cpu usage issues when worker count increases?
Mubeen Ali hey Mubeen, from reading online about greenlets( first time I know about if) it seems its just multithreading or multiprocessing system for a single node) so I don’t think it give the power to scale horizontally like a fully fledged message queue system like rabbitMQ
Again this is perfectly fine on a small system and you can go very far with a single instance with multi processing) keep it simple is my moto
@@hnasr thanks for the openion. now i can work on more options
Seen it been use a lot for change data capture on etl data engineering side of things
Great video. FYI There is a low hum on your audio.
great video hussein, can you please explain to me the difference between message queues and message broker
wassim queues are the actual data structure, broker is the queue manager. Clients talk to message broker that writes and reads from the queue
@@hnasrunderstood! thanks hussein
But how will the client on frontend find out that computation is done ? Should client do every x seconds GET request to get actual state ?
Your could have explained using block diagram, leaving the video in between
Great. Wondering, when to go for MQ's like AMQ, Rabbit vs Kafka?
Good rule of thumb is when requests are naturally long running for synchronous workload. I gave example with compression or copyright check
@@hnasr Thanks. I mean for async applications, how to choose MQ like between RabbitMQ vs Kafka. If you have covered this already in a different video, I can check that out.
Great video. But according to your definitions, Kafka is more like a pub-sub system than a queue.
Daniel Fazeres Correct Kafka is a Pub/Sub system that can be configured to act like a queue with consumer groups. RabbitMQ is a message queue that can be configured to be a pub/sub system with exchanges.
I am stuck at last part you talking about web server doesn't calculate anything, its just response to incoming request but if web server did not calculate who do that :) I know its weird question but im not a experienced developer.
No it is actually a very important question! Another service can be responsible to do the actual work. You make a request, the web server accepts it and writes it to the queue, and responds back to the client. A service talks to the queue and consume the work and do the heavy work and takes 1 hour or so and writes it back to the queue.
Here is an example Illustrated here Publish-Subscribe Architecture (Explained by Example)
th-cam.com/video/O1PgqUqZKTA/w-d-xo.html
That being said I if your requests can be processed quick your webserver can server that traffic. Its all a game of trade off. Only use a tech when you absolutely need it
@@hnasr Thank you for good explanation, i have project ideas needs to has multi services, they are not gonna be a simple crud aplication so your videos will give me aspirations to build that projects, thank you again.
It is wonderful if you believe that all technology exists for a reason. Ever heard about the not-invented-here probloblem?
Your explanation is very good but hard to visualize. It would be great if you can write or draw as you speak.
Should be also in the message queue playlist?
Thanks for helping me organize the content 😊
@@hnasr no problemos
Helps me, helps you. Something small that I can do to thank for your great investment
Why would you want to use a queue if you are using serverless (i.e. Cloudflare Workers with Cloudflare Queues)
"Request is indeterministic" and "process is by nature long running"
Doesn't these two points eventually mean same thing?
Kafka is more pub/sub streaming ('reactive') than a queue like AWS SQS
Hey man, What's with you and reverse proxy ? Lol, been hearing that phrase in almost all the videos . JK :)
Nice video. I wish you used diagrams to explain the concepts rather than waving your hands.
🌆8pm july 4 2022
6:44 solution > inform user of a received request using queue, then alert user the request is successful; basically pretend its done while it actually not
It uses amqp protocol.
he seems like a modern Beakman lol