Men! I can't appreciate all your videos enough. They had guided me on from zero to "hero" on my serverless journey in my new job (I started 6 months ago). I have a whole notification system for AWS Backup using SNS / SQS / Lambda running right now!
You really don't understand how this video just saved me stress. Thank you. Love the visual explanations and the details such as common use cases, etc... keep it up
I was always scared of the SQS/SNS, and could never really understood them😅. Thank to your videos I do now! Very grateful for your great content and very neat and clear explanation, thank you a lot!☺
Thanks a TON for such a tremendous , easy to digest video. Quiet explanatory at the same time. LOVED your pace of teaching and explaining the concepts. Stay Blessed Brother !
Indeed the video is very informative ,Thanks for sharing . I have a small doubt ,similar to query parameters in API how we incrementally poll messages from SQS queue. I would need to poll messages based on a particular timestamp.
Great Explanation. Have a quick question. I have multiple queues where the poller should be fetching the messages from all of them based on some priority. And the priority would be like this - for example 1st queue having only 2 messages, 2nd queue having 100 messages and 3rd queue having 1000 messages then I want the poller to initially poll the queue which has less number of messages i.e here the poller should initially poll 1st queue then the 2nd and then the 3rd. How can this be achieved.
You are AMAZING!!! I've not had a single video explain anything as clearly as you do. Please keep doing what you're doing and MORE of it, because you are a natural teacher of the best kind.
What happens in a FIFO if a consumer could not correctly handle a message and it is requeued? Does it block other messages being processed, since it's a FIFO?
Hi B R, great question. I couldn't find any direct answer to this on the AWS documentation but here is something insightful. From here, under "FIFO queues": docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html "FIFO queues provide exactly-once processing by consuming messages in sequence from a message group. Thus, although the consumer can continue to retrieve ordered messages from another message group, the first message group remains unavailable until the message blocking the queue is processed successfully." I feel like based on this it probably blocks the queue from processing messages in the same message group. However to know for sure its probably worth a quick experiment.
@@BeABetterDev I played around with it and FIFO-s are blocking. Inserted 3 messages into a FIFO with send-message, then the first receive-message returned the first inserted one, but then calling receive-message again didn't return anything. Had to do a delete-message with the receipthadle of the first message to get the second one inserted. You'd better handle your exceptions in the code that processes the messages.
Thanks for this great overview! Quick question, as part of the "at least once delivery" that standard SQS provide, can I assume that duplicates will have the same messageID? Also, even though it is "at least once delivery", I can assume that there won't be two versions of the same message in a queue at the same time, thus respecting the visiblity timeout you mentioned at 11:20, right?
Nice. My question is: what are the drawbacks of this continual "polling" being done by Service B in your minute 17:30 example? Suppose your service A does not send messages to Service B for months on end, is this "polling" costing anything (i.e., computational cost in terms of site performance, AWS fees)?
Should have finished the video before asking... it seems like Long Polling avoids the need to continually poll for messages. Still curious about the drawbacks of this, though.
Great video. Quick question: when you mention cross account patterns, do they also support cross partition data transfer? For example if I have an AWS account in China and another in USE1, can I publish from my SNS queue in China into my SQS queue in USE1?
Hi Bijan, Cross region sns to sqs works for all cases except involving the china region. I found this post on the topic if you're interested: stackoverflow.com/questions/69061869/is-it-possible-to-publish-sns-messages-from-global-aws-account-to-sqs-queues-in Hope this helps
@@BeABetterDev LOL, thanks heaps for checking it out. Yeah, unfortunately it seems it is not possible. I am investigating if there is a way to make that happen; IAM seems to be out of the picture; I am thinking of using temporary security credentials with SQS: docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-using-temporary-security-credentials.html ; will keep you posted if I could somehow pull it off
Thanks for all these videos. Do you have a video on local dev environment with aws - I couldn't find one? How do you approach this? Do you test against another dev aws environment or do you spin up everything locally with mocked/faked services like localstack?
Hi Potato Man! Check out this video on my setup: th-cam.com/video/CGCn0b4FOfs/w-d-xo.html I haven't made one for IDE / plugins but if you think it would be useful I can make one. Cheers
Great video. What are the downsides/tradeoffs of using SQS vs. SNS? In my mind SNS can do everything SQS does and more (i.e. supports multiple subscribers). Also you mention that the consumer sets the consumption rate. How would that work with something like having a Lambda as the consumer? I'd imagine Lambda would spin up as many instances are required to process as many SQS messages as quickly as possible.
Hi @BeABetterDev , 1). Can SQS accept an Event Published from a System (example : Salesforce - Events) ? 2). What is the count of messages that an SQS can accept from an external System (i.e, Published Message Count from System -> Amazon_SQS ) ? 3). Can SQS accept Bulk Messages / Events ? Please let me know. Great video, Clarified in a single go.
Lovely AWS tutorials. Kudos to you and I hope you will keep it going! Could you please better explain the beginning? 00:00:42.160 in the past when two services had to 00:00:44.399 communicate with one another one service 00:00:46.559 would perform an api call to inform the 00:00:48.960 other service that something changed or 00:00:51.039 some kind of event occurred on its side 00:00:53.680 now using sqs this contract has changed 00:00:57.039 instead the producing service of the 00:00:59.359 event or the notification can 00:01:01.359 asynchronously notify the consumer that 00:01:04.080 something changed in its system I do not understand what you were trying to imply.. by how I interpret it & how the above text is phrased; you say - in the past: one service had to inform the other something changed on its end through an API call - now: the contract changed, id: the service in which something changed (same as above till now) can asynchronously notify the consumer that something changed in its system... how do the above differ? did you mean API calls in the past were always "blocking" instead of "async"? is this what you meant? I do not follow, sorry..
PURE GOLD. Thanks a loooooot Be A Better Dev is my shepherd; I shall not want. He maketh me to lie down in green pastures: he leadeth me beside the still waters. He restoreth my soul: he leadeth me in the paths of righteousness for his name's sake.
Hey Better dev, Just wanted to thank you for the tremendous time you spend on putting all these materials together and then compiling them for the great and educative videos you produce for all. You are a true leader, a role model, and a trendsetter for many. Keep up the good work because you are impacting lives worldwide, and I want to thank you for your selfless sacrifice to the human race. Jah, Rastafari!!!
Your AWS videos are far better than paid courses. Very clear explanation. Thank You!
Men! I can't appreciate all your videos enough. They had guided me on from zero to "hero" on my serverless journey in my new job (I started 6 months ago). I have a whole notification system for AWS Backup using SNS / SQS / Lambda running right now!
You really don't understand how this video just saved me stress. Thank you. Love the visual explanations and the details such as common use cases, etc... keep it up
Thanks so much Jude and glad you found it helpful!
I was always scared of the SQS/SNS, and could never really understood them😅. Thank to your videos I do now! Very grateful for your great content and very neat and clear explanation, thank you a lot!☺
Same same, it all sounded to me like some sort of witchcraft, but now i get it 😂
Thanks a TON for such a tremendous , easy to digest video. Quiet explanatory at the same time. LOVED your pace of teaching and explaining the concepts.
Stay Blessed Brother !
Glad the video helped and thank you so much for the kind words!
This video has been extremely helpful with my studies for SAA, just wanted to thank you so much for this
Glad it was helpful!
one of the best explanation out there and it's free. Good job man
Glad you liked it!
I think this is the best channel to get a great overview of AWS.
Another clear, straightforward, awesome video!! Thanks, man!!
You're very welcome Clerton!
Excellent introduction to SQS! Thank you!
Excellent video as always!
thanks for putting all this information short for us, lazy ones
Thoroughly appreciate this video. You have no idea
Awesome content…saved me hours of google and reading through documentation 👌🏽
Glad it helped!
Great illustration, thanks for your efforts. Much appreciated!
great, thanks for explaining it in a really simple way
Thanks bro, nice video. Keep it up.
You're very welcome and thank you!
Indeed the video is very informative ,Thanks for sharing . I have a small doubt ,similar to query parameters in API how we incrementally poll messages from SQS queue. I would need to poll messages based on a particular timestamp.
Sqs has a limit of 10 messages and 256kb message size. That sounds something very obsolete technology not really modern high performance
very well explained. Thanks!
Top notch content man. Thanks!
Thank you for this fancy explanation
Amazing tutorial loved it
Thanks
Thank you so much for your generosity!!!
Amazing video
Great Explanation. Have a quick question. I have multiple queues where the poller should be fetching the messages from all of them based on some priority. And the priority would be like this - for example 1st queue having only 2 messages, 2nd queue having 100 messages and 3rd queue having 1000 messages then I want the poller to initially poll the queue which has less number of messages i.e here the poller should initially poll 1st queue then the 2nd and then the 3rd. How can this be achieved.
You are AMAZING!!! I've not had a single video explain anything as clearly as you do. Please keep doing what you're doing and MORE of it, because you are a natural teacher of the best kind.
Very Good explanation, i like it👌
Thank you so much
What happens in a FIFO if a consumer could not correctly handle a message and it is requeued? Does it block other messages being processed, since it's a FIFO?
Hi B R, great question. I couldn't find any direct answer to this on the AWS documentation but here is something insightful.
From here, under "FIFO queues":
docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
"FIFO queues provide exactly-once processing by consuming messages in sequence from a message group. Thus, although the consumer can continue to retrieve ordered messages from another message group, the first message group remains unavailable until the message blocking the queue is processed successfully."
I feel like based on this it probably blocks the queue from processing messages in the same message group. However to know for sure its probably worth a quick experiment.
@@BeABetterDev I played around with it and FIFO-s are blocking. Inserted 3 messages into a FIFO with send-message, then the first receive-message returned the first inserted one, but then calling receive-message again didn't return anything. Had to do a delete-message with the receipthadle of the first message to get the second one inserted.
You'd better handle your exceptions in the code that processes the messages.
thank you for this video!
thank you so much for this
Why sending messages to Queue is an anti-pattern?
Thanks for this great overview! Quick question, as part of the "at least once delivery" that standard SQS provide, can I assume that duplicates will have the same messageID?
Also, even though it is "at least once delivery", I can assume that there won't be two versions of the same message in a queue at the same time, thus respecting the visiblity timeout you mentioned at 11:20, right?
Perfect explanation
Glad it was helpful!
Would this be ideal for returning a dictionary from a lambda as a message to be consumed by other lambdas?
25:24 I guess Clodudwatch is for monitoring failures to launch (duds)
man ! i wish i work under you as my senior !
One day ! :)
@@BeABetterDev thx
Nice. My question is: what are the drawbacks of this continual "polling" being done by Service B in your minute 17:30 example? Suppose your service A does not send messages to Service B for months on end, is this "polling" costing anything (i.e., computational cost in terms of site performance, AWS fees)?
Should have finished the video before asking... it seems like Long Polling avoids the need to continually poll for messages. Still curious about the drawbacks of this, though.
Great video. Quick question: when you mention cross account patterns, do they also support cross partition data transfer? For example if I have an AWS account in China and another in USE1, can I publish from my SNS queue in China into my SQS queue in USE1?
Hi Bijan,
Cross region sns to sqs works for all cases except involving the china region. I found this post on the topic if you're interested: stackoverflow.com/questions/69061869/is-it-possible-to-publish-sns-messages-from-global-aws-account-to-sqs-queues-in
Hope this helps
@@BeABetterDev LOL, thanks heaps for checking it out. Yeah, unfortunately it seems it is not possible. I am investigating if there is a way to make that happen; IAM seems to be out of the picture; I am thinking of using temporary security credentials with SQS: docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-using-temporary-security-credentials.html ; will keep you posted if I could somehow pull it off
Thanks for all these videos. Do you have a video on local dev environment with aws - I couldn't find one? How do you approach this? Do you test against another dev aws environment or do you spin up everything locally with mocked/faked services like localstack?
Hi Potato Man! Check out this video on my setup: th-cam.com/video/CGCn0b4FOfs/w-d-xo.html
I haven't made one for IDE / plugins but if you think it would be useful I can make one.
Cheers
fucking legend.
Can we do an interview, need you input for my university project
Please brother
Your videos are amazing! Thank you so much!
You're very welcome Semyon!
👍
What if the Queue goes down lol
Good, you mispronounce the word 'process' in all of your videos. It's a little distracting but the content is excellent👍
Great video. What are the downsides/tradeoffs of using SQS vs. SNS? In my mind SNS can do everything SQS does and more (i.e. supports multiple subscribers).
Also you mention that the consumer sets the consumption rate. How would that work with something like having a Lambda as the consumer? I'd imagine Lambda would spin up as many instances are required to process as many SQS messages as quickly as possible.
Hi Jean, I have a video that compares the two check out the link below. th-cam.com/video/mXk0MNjlO7A/w-d-xo.html
Hi @BeABetterDev ,
1). Can SQS accept an Event Published from a System (example : Salesforce - Events) ?
2). What is the count of messages that an SQS can accept from an external System (i.e, Published Message Count from System -> Amazon_SQS ) ?
3). Can SQS accept Bulk Messages / Events ?
Please let me know.
Great video, Clarified in a single go.
U R A WONDER
Superb. The clearest description of SQS that I've come across.
Lovely AWS tutorials. Kudos to you and I hope you will keep it going!
Could you please better explain the beginning?
00:00:42.160 in the past when two services had to
00:00:44.399 communicate with one another one service
00:00:46.559 would perform an api call to inform the
00:00:48.960 other service that something changed or
00:00:51.039 some kind of event occurred on its side
00:00:53.680 now using sqs this contract has changed
00:00:57.039 instead the producing service of the
00:00:59.359 event or the notification can
00:01:01.359 asynchronously notify the consumer that
00:01:04.080 something changed in its system
I do not understand what you were trying to imply.. by how I interpret it & how the above text is phrased; you say
- in the past: one service had to inform the other something changed on its end through an API call
- now: the contract changed, id: the service in which something changed (same as above till now) can asynchronously notify the consumer that something changed in its system...
how do the above differ? did you mean API calls in the past were always "blocking" instead of "async"? is this what you meant? I do not follow, sorry..
Excellent introduction to SQS!
Thanks so much Mark!
Elegant way of explaining - simple and crisp - no fancy show business. Great job!
Thanks!
Simply explained, thank you !
Keep up ;)
You're welcome!
Gratitude
You're a great teacher - thank you for the video
Thanks so much Andrew! Glad you enjoyed :)
Thanks for sharing comprehensive understanding of SNS. Appreciate your efforts!!
PURE GOLD. Thanks a loooooot
Be A Better Dev is my shepherd; I shall not want. He maketh me to lie down in green pastures: he leadeth me beside the still waters. He restoreth my soul: he leadeth me in the paths of righteousness for his name's sake.
Excellent introduction to SQS! Thank You!
Thank you
You're welcome!
Hey Better dev,
Just wanted to thank you for the tremendous time you spend on putting all these materials together and then compiling them for the great and educative videos you produce for all. You are a true leader, a role model, and a trendsetter for many. Keep up the good work because you are impacting lives worldwide, and I want to thank you for your selfless sacrifice to the human race. Jah, Rastafari!!!
Thanks for this video!! Keep it up!!
Very Clear Explanation!
This is the best tutorial video ive seen for SQS
Great video !
Glad you enjoyed it!
Great explanation thanks!
Thank you very much
Legend !!