How to connect a Lambda function to SNS using an SQS queue

In this video you will learn how to use SQS as a buffer, throttle, or automatic retry-mechanism between an SNS topic and a Lambda function. Full playlist: • Hands-on deep-dive int...
TIMESTAMPS:
0:00 - Why use SQS to connect SNS with a Lambda function?
0:41 - Initial setup: A lambda function triggered by SNS
1:18 - Testing the initial setup without automatic retries
2:13 - Create an SQS queue
2:31 - Subscribe SQS queue to an SNS topic
2:46 - Rewire the SNS topic and the Lambda function to use the SQS queue
3:27 - Necessary permissions for Lambda to receive messages from an SQS queue
5:04 - Testing the new setup with automatic retries
#AWS #AWSinFiveMinutesOrLess #Serverless

[@ranvijaymehta]

thank you so much sir

[@ranvijaymehta]

Love from Bangalore

[@lavakumar5181]

Awesome content...thanks

[@DTraub]

Thanks!

[@baptista21]

Is it possible with sns sqs and lambda to handle an external service that supports 1 call at a time, that is, my asynchronous messages become synchronous so that once my service responds, it executes the following message?

[@DTraub]

I'm not entirely sure what you mean, so I'll explain two different ways below.
1. An external application polls new messages one by one from the SQS queue.
2. SQS posts every new message to an external endpoint and waits until it's available again.

[@DTraub]

1. An external application polls new messages one by one from the SQS queue:
Any application that has access to the internet can access an SQS queue if it has AWS credentials (Access Key ID and Secret Access Key) for a user or role with the required permissions.
Note that manual processing requires the caller to take care the deletion of the message after successful processing, otherwise it will reappear after the visibility timeout (Learn more about the visibility timeout, how it works, and how it can be changed here: docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html)
I've uploaded a python sample that demonstrates the flow here: gist.github.com/DennisTraub/308d27b583249e75956148a083f8573a

[@baptista21]

@@DTraub I tried to make an example with an api that had a delay of 10 seconds so I sent 10 messages to sns and my sqs was subscribing to the topic, then I had a lambda that called the api that was tagged every time a sqs message entered, I configured this lambda with concurency 1, but the problem I had when doing the test was that the messages were lost, some of then not all maybe was for the visibility timeout

[@DTraub]

2. SQS 'posts' every new message to an external endpoint and waits until it's available again.
Unfortunately, SQS cannot publish messages. SQS queues can only store messages so they can be polled by other services or external applications.
This means that you need some kind of process that polls messages from the queue, and posts them to the downstream service one by one.
One option could be a Lambda function that takes the individual messages from SQS and publishes them to an SNS topic, which in turn posts the messages to a customer-managed downstream endpoint.
I've uploaded some example function code in python that shows how this can be done: gist.github.com/DennisTraub/a5c977cf2ec7cf207ad008dcdf6b645c
Note that SNS will automatically retry posting the message to the the customer-managed endpoint in regular intervals for a maximum of 3600 seconds (1 hour) if the endpoint is not available. Learn more about SNS delivery retries here: docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html
You can also add a Dead-Letter Queue (DLQ) for messages that can't be delivered within that timeframe. Learn more about SNS DLQs here: docs.aws.amazon.com/sns/latest/dg/sns-dead-letter-queues.html
I hope this was helpful.

[@baptista21]

@@DTraub I think I found the solution with an aws tool called bda kinesis, kinesis can store the sns messages, and kinesis later you can handle them synchronously, for example in a lambda function it can be called every time a kinesis message comes in, so Your lambda function waits for each message to be processed before executing the next one.

[@baptista21]

Do you have the lambda function which randomly fails?

[@DTraub]

Sure. I've uploaded the function code to GitHub. It's able to process SNS and SQS messages, but is set to fail 50% of the time: gist.github.com/DennisTraub/b339041784a91db4a18dcee8103ce6ec