AWS SQS to Lambda Tutorial in NodeJS | Step by Step
ฝัง
- เผยแพร่เมื่อ 7 ส.ค. 2019
- SQS integration with Lambda functions are a new offering from AWS that solve many of the problems of SNS + Lambda. In this video, I show you exactly how to set up a SQS queue and consume events from a Lambda function. I start out by talking about some of the basic concepts of SQS and Lambda integration. From there, I walk you through the AWS console and show you how to create a SQS queue and a Lambda function. Additionally, I show you which permissions you need add to your Lambda IAM role. Finally, I test the setup by manually pushing messages to the Queue and verify they are being processed in the Lambda function.
Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! courses.beabetterdev.com/cour...
🎉SUPPORT BE A BETTER DEV🎉
Become a Patron: / beabetterdev
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driver Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
Resources:
SNS to SQS Setup: • AWS SNS to SQS Tutoria...
AWS Lambda Introduction: • AWS Lambda Introduction
Getting started with AWS: • Introduction to AWS | ...
☁Topics covered include:Lambda Concurrency Gotchas
- Lambda SQS trigger
- Lambda DLQ
- Lambda Concurrency
- Lambda / SQS Permissions
- SQS Visibility Timeout
- SQS Message Retention Period
- SQS Receive Message Wait Time (aka Long Polling timeout)
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
#Serverless
#SQS
#SimpleQueueService
#Lambda
#AWS
![[LIVE] : ONE ลุมพินี 70 | คู่เอก "โฟกัส vs สเตฟาน"](http://i.ytimg.com/vi/_fVieQ89DRQ/mqdefault.jpg)
![fellow fellow - Proud [OFFICIAL MV]](http://i.ytimg.com/vi/9uZF7BlUQPE/mqdefault.jpg)
Great video! Also no need to apologize when you make a mistake, love seeing videos that include the mistakes in them. It is actually helpful!
Thanks Paul! I can't help saying sorry sometimes :)
@@BeABetterDev You are Awesome guy
I love videos like this. HOURS of lecture about SQS and I don't get "long polling." You explain it in about 5-10 seconds. "Ask less frequently, 'Hey, you have anything?'" The object being to get fewer instances of "Nope, nothing" from the queue, which also means the Lambda is firing less often -> less $ spent for little gain. This rocks!
So glad you found it helpful! Welcome to the channel!
but the only challenge I see there is a concurrency rate would be decreased?
I have been doing only frontend tasks so far and now my manager told to start touching backend also. At first it looked very complicated and frightening, but your videos slowly sort everything out. You channel is the gem, thanks a lot. You have very good explanation skills.
perfect to the point walkthrough. exactly what I was looking for...
Thanks Mark! Glad you enjoyed :)
you got me when you forgot to save the lambda, nice video, thanks.
Such an amazingly clean explanation! Liked and subbed!
Thanks Mitlandir! Welcome :)
Thank you very much for this video Daniel! It was very helpful to me!
Very nicely presented, Thanks
Great video and a lovely explanation.
thanks for the tutorial! very clear
very welcome!
Like the way of explanation! Thank you!
Glad it was helpful!!
Awesome tutorial! Very very nice!
You're very welcome Rafagan!
Awesome tutorial!
Thanks Daniel! Glad you enjoyed.
I enjoyed the tutorial.
Very nice presentation. Thanks for sharing.
Thanks for your comment!
Great video thank you
Great video, Thanks !!!
You're very welcome!
Thank you.
Great video. thanks
Very welcome
Great video
Glad you enjoyed it
finished watching
Awesome
you are a saviour
Thanks Aniket!
Very Intersting tutorial thanks! Regarding batch messages apparently if one fails the entire batch then has to retry and is not clear which message has failed so generally I prefer to set that parameter to 1.
Thanks Gregorio! I share the same sentiment regarding batch processing. In my view there isn't enough benefit to performing batch processing that outweighs the cost of partial failure scenarios.
@@BeABetterDev And worthy to note the maximum batch size is 10 and I think higher the batch value translate to less lambda cost because lambda function will pull 10 messages with a single action. so less spin up more message receives.
new subscriber here :) great video!!
Thanks so much Muhammad and welcome to the channel!
Very nicely explained.. Waiting for SES+Lambda..if possible customize template.. Thanks for sharing your valuable knowledge..
Thanks Vishal!
Awesome video!! It worked smoothly for me.
One doubt though: in minute 23:32 when you have your lambda ready, you have two destinations: Amazon SQS and Amazon CloudWatch Logs. I have none of those on my Lambda. How do you set that???
Hey there... Awesome and simple video!! Do you know by any chance if there is a good wrapper in node to push messages to SQS?
Hi Veronica,
Thank you! I did a quick search and found that the aws-sdk npm repository which contains a client wrapper you can use with SQS.
The node repo is here: www.npmjs.com/package/aws-sdk?S_TACT=105AGX63
And an example on how to use it to push to SQS is here: docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/sqs-examples-send-receive-messages.html
Hope this helps and thanks for watching!
Well done. (If I had a dollar every time I forgot SAVE... )
Thanks Glenn!
well explained...
can I use SQS in my use case...like
I have a requirement to auto start an EC2 when a user got uploaded file in S3. for this I have created Lambd function and add a S3 event notification trigger in lambda...it is working fine, and auto starting as well and with a basic function I'm stoping EC2 when the instance process got finished.
But the problem is...before stoping EC2 if user upload other file in s3...I want notification sould hold till EC2 stop, once
the EC2 stoped second notification will pass to lambda then EC2 starts again...
Hi Nice explanation
I am facing one issue.my lambda is invoking only for the first message pushed to the queue but for second time it does not invoke.I have used batchSize 1 .Please suggest where I am doing wrong.Thanks
Please do on batch response failure and Dead-letter queue
Hello, Dan! First of all thanks for posting this tutorial.
In the section about ‘Default Visibility Timeout,’ you stated that the consumer should manually delete the messages. However, in the example provided, the messages are being automatically deleted. Our consumer (in this case, a Lambda function) is not handling message deletion. I’m confused.
❣
Thank you so much for the video. Does the Message get automatically deleted from Queue without having to this manually from Lambda after it has been triggered?
Hi Riham, it automatically gets deleted as long as your function call returns successfully. Hope this helps.
Hi, thanks for the great video. One thing i wanted to know that when tried sending the message using SNS to SQS. I also attached a lambda function to read data from SQS. In the monitoring tab where the graph is shown, i saw that the graph of 'Number Of Messages Sent' started before the graph of 'Number Of Messages Received'. I thought message should be received first and then sent. Correct me please if i am wrong and please explain.
Hi Chandan,
I found this article that may be of help to you explaining the differences between these metrics. Take a look: www.bluematador.com/blog/how-to-monitor-amazon-sqs-with-cloudwatch
Hey brother, excellent tutorial, I'm very thanksfull with you!!! I have a question about lambda Poller, What happens if at end of day my application dosen't send any messages to SQS but mi lambda poller continue asking for any messages. Does it have any cost? ... thanks on advance!
Great...just one question. Is it possible to get a specific event from SQS, like deletion for example?
You should use a SNS queue instead for that and subscribe to topics
Do you have to use a Lambda function to read and delete messages from SQS or is there a step function defined to do this? I already use step functions in my solution and currently do not need to use a Lambda function. I see there are SQS support in step functions but I don't see any documentation of video explaining how to use them. Thank you.
Does AWS charge for the concurrency run-time? Seeing that it could be set at a specific limit.
Hi great video.
I am hoping you can comment on my use case which is similar but not quite.
I have a client that pushes message to SQS, I have Alexa trigger a lamba which I want to poll the queue and then it will speak the message. So the main difference is ASK is the trigger for the lambda not SQS and the lamda performs a poll at the time it is triggered.
Cheers
Hi Paul. This sounds like a reasonable application of SQS + Lambda. Sounds like a neat project!
I would like to limit the SQS to invoke a lambda at a certain rate, a maximum 500 times per second. Anyone have an idea how to do that? Does SQS have a setting to slow down? : )
in the lambda function, I don't see you delete the message. How did it get deleted? is there a setting to delete after reading the message?
Can you tell how to trigger a lambda using sqs , all in the cloudformation template?
What's the purpose of using Boto3 here.Cannot the other lambda be directly called from the invoker as both of them are on AWS?
Hi,
I am new in aws . i have some question which mention below
1. I am using sqs fifo queue to receiving the data into the node app. Without lambda function i can't trigger the receving message continuously into the node application.
2. If we need lambda function for trigger the queue and receving the reposone in to the node application.
3 . How can i continuously consume the sqs queue in to the node application
thanks
I am facing issue with the api gateway integration with SQS. can you please help me with it. I want to write the serverless.yml mentioning the integration of api gateway with SQS. (in this case while creating the API GATEWAY THE CODE SHOULD BE SUCH WHICH SHOULD HAVE INTEGRATION TYPE AS SQS).
Please let me know how this can be done. ?????
I was able to receive message ID but I am not able to view the contents and attributes of the message in PYTHON. can you pls help?
Splendid 👌 could you provide less step by step and more theoretical lectures on AWS services
Thanks for the feedback Tomer. I'm currently trying to mix it up but expect a new overview lecture this weekend on Cloudwatch Anomaly Detection!
How is it possible to do the same thing with FIFO queues? Thanks
aws lambda not supporting FIFO queues for now check their docs
khalid is correct - FIFO queues are not supported for AWS Lambda Triggers.
See documentation here: docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-lambda-function-trigger.html
"Note: FIFO queues don't support Lambda function triggers."
Fifo is now officially supported for lambda!
I am not able to understand what does the term "lambda vacation" mean? Could you please help to understand the term
Thanks this is a great vid. But I hope someone can help me with the following problem. The Lambda will process the message and then wants to pass the message onto another AWS service. At the moment I have to hard code the properties needed to call the other service. It can be any service S3 bucket or. ElasticSearch. But I would rather have those details set up in a separate config file, similar to properties or a yml file, on a server based application. Is there a way to do this. Thanks.
Hey Tony,
You can use something called Lambda Environment Variables. These can be fed in dynamically as config values for your function. Alternatively, if you're looking for a full-fledged config service, there's AWS AppConfig.
Hope this helps
Daniel
@@BeABetterDev Thank you, I did actually find AWS AppConfig in my research. I will do a review on AWS AppConfig vs Lambda Environment Variables, and will see which is best for my needs. Again thank you.
Hey can I send message from AWS lambda to fifo sqs?
Hi Ajay, absolutely you can!
I am looking for lambda code in python, where it accepts an sns topic via sqs and then triggers a step function, any suggestion where I can find such thing in python ? One thing to mention, I have to do everything by python code and can't do anything via console. I will be creating resources via terraform
Hi there,
I don't have a specific resource for this kind of functionality. However you may be able to piece together some of my smaller videos to deliver this functionality. Good luck!
where do I visualize that the lambda is polling SQS for new message? I can't see any log for that.
Thank you so much!
It's very good video everything is detailed but I am following this video and not getting SQS inside this lamda function timestamp: 22:00
Could you please give a video of the SQS to Lambda scaling issue? We are facing the issue that if SQS has a spike, and there isn't enough Lambda, after several retries, messages from SQS will be sent to DLQ.
Hey G O - can you tell me more about your problem? Are you seeing throttling on your Lambda function?
@@BeABetterDev yes. Basically the root cause is, the SQS to Lambda trigger works in a way that there are workers pull messages from SQS regardless of whether Lambda is throttled. If Lambda is throttle, then the message just go back to the SQS after the visibility. If the same message is picked up more than a certain time, it will be throw to the DLQ.
@@BeABetterDev There are ways to mitigate this issue, like increase Lambda quota, but no permanent fix. The only way to fix it is to not use the trigger, but write our own logic to pull message from Lambda. However it is hard to tell how many lambda I should run.
Hi G O,
This is a very interesting problem that only occurs at the edges of scale. Have you tried looking into using Provisioned Concurrency with an AutoScaling policy that tracks the # of messages in your queue? This way, you can ensure that you always have enough capacity to handle the workload and can control the amount of processing power you have through a managed policy.
Hope this helps!
@@BeABetterDev Hi, thanks for the reply. After the burst, Lambda can only scale up at the rate of 500 more instances per minute. If request rate is more than this, the throttle can still happen. So long as there is throttle, there is chance that message will be received multiple times and end up in being sent to DLQ.
Hi, This is a great video. But I hope someone could answer the following question:
My AWS lambda needs to establish a database connection and serve the requests. And if there's a big spike of traffic then AWS lambda might spins up 1000 containers to serve the requests(Ex: If there are 20000-30000 sqs messages), but if database only allows, say, 100 connections then most of the requests won’t be able to connect to the DB. Then what should be the batch size and reserved concurrency be initialized to ??
Hi Mike,
Do you happen to be using Aurora? If so you can use RDS Proxy which manages the connection pool for you.
If not, you'll probably need to limit the reserved concurrency to the maximum number of concurrent connections you can support.
Hope this helps
@@BeABetterDev Thanks for your reply!
Can I send food in a queue? So like, when I'm hungry, I just request a sandwich and it pops out?
Haha, I wish it was that simple!
Your fridge is kind of like that, right?
Is there a way to configure the lambda so doesn't do anything with the message when its triggered? I'd like to set something up so that the lambda gets notified that there is a message in the queue, and then spins up an EC2 instance to actually pull and process the message.
Thanks!
Hi Wayde,
You can certainly just ignore the contents of the message and spin up your EC2.
Another way of doing this is to set up an Autoscaling group that triggers based on the SQS metric - you can set it up so that it will automatically spin up an instance when queue size > 1.
Hope this helps and thanks for watching!
"message": "Endpoint request timed out" :(
Is your Lambda function performing a job that takes longer than 3 seconds? My default 3s is the Lambda timeout duration. You can adjust this in the configuration section of your function.
If you can trigger a Lambda by SQS, why you need to poll SQS. I think you need to publish new video :P AWS is changing
shouldn't you delete the messge?
Hi John,
If you return out of your lambda function without throwing an exception, Lambda handles deleting the message for you!
Not sure what you or I can do about it but the fuzziness of the aws window text is head aching, my only complaint about your otherwise great videos.
Thanks Tom! This video was one of my first on TH-cam. Since then, I've made changes to the way I record. If you check out my more recent videos the quality (both video and audio) is much better.
Thanks again for watching and your feedback
I said it "u didn't press save" then I realized it was a recorded video from years ago..
One of the simplest tutorial I have ever seen. 2 min is actual execution time , rest is all explanations. I was hoping for more advanced implementation or a practical demo using an application flow logic.
Not sure if this is a compliment or criticism, but thank you for the comment. In the future I will include more advanced implementation videos. Thanks.
The Tutorial should be in Python and you have used NodeJS for Lambda Creation. its confusing!!!
Apologies for the confusion - I updated the title accordingly.
in the lambda function, I don't see you delete the message. How did it get deleted? is there a setting to delete after reading the message?
Hi Mahadevan,
Deletion of the message happens automatically by Lambda on successful process of the message. That is, if your Lambda does not throw an exception and returns normally, the message is considered successfully processed and followed by a SQS deletion.
Hope this helps clarify,
Daniel