Hey, great video! One query: when SQS triggers a lambda function, that is a SYNC call, correct? Then why are we modifying the "Asynchronous invocation" in the lambda func config? The retry mechanism and DLQ service in lambda are irrelevant and without doing that too, we can send our messages from lambda-queue->lambda_func->lambda-dlq
That's true. The retry that actually worked for him seems to be from the lambda-queue configuration. Without adding lambda-dlq as a DLQ for lambda-queue, it would not have worked.
@@SchoolScience You are absolutely right. What is being displayed in the video is not wrong but, imo, misleading. The Lambda DLQ setup here is not needed. You can either do: 1) Queue to DLQ setup This is essentially what the video shows. Create two queues, set one queue to be the DLQ, create a lambda trigger but DO NOT set a DLQ config in lambda. Post a message to the queue and check that it ends up in the DLQ 2) Lambda to DLQ setup Create a queue and set the lambda DLQ config to be the queue you just created. Now invoke you lambda asynchronously (for example by creating another lambda that invokes your "DLQed" lambda with type "Event"). Notice that your "DLQed" lambda failed invocations now end up in your queue WITHOUT having to setup a second queue. Hope this helps whoever is struggling with this as I was
Great Video Soumil, straight to the point. I have a question. I am trying to use SNS FIFO --> SQS FIFO --> lambda function for processing. SNS and SQS are Content-based deduplication enabled. If i pass one record working fine but if i send multiple producer messages, one record being processed rest are being deleting. Any help?
@@SoumilShah thanks for the response. In simple, SQS FIFO triggers lambda function. SQS sent 4 messages but lambda invoked twice. Concurrency is "Unreserved account concurrency & 1000". What can go wrong?
@@anil1970 the reason lambda is invoked twice data is sent in batches in lambda lambda one got 2 records and lambda 2 got 2 records Also depends on if you are using king or short polling and what batch size window you are using !
@@anil1970 what is your ask do you want 10 messages is that what you want ? Increas delay in sqs queue Delay is time after which message appears in sqs
I have a question, if I don't want my entire batch to get rejected in case there is an error in specific record. Can I push those particular records to the DLQ using boto3 calls instead of configuring DLQ on lambda with retey counts which will eventually rejects the entire batch or it is not a good design practice?
Yes but that would be issue The term used for that is poison messages Usually poison messages are better handelled in kinesis then dlq yes if you have try catch and if you just want to publish that message you can add that in catch blok )
Could you slow down and move the mouse only when necessary please
Hey, great video! One query: when SQS triggers a lambda function, that is a SYNC call, correct? Then why are we modifying the "Asynchronous invocation" in the lambda func config? The retry mechanism and DLQ service in lambda are irrelevant and without doing that too, we can send our messages from lambda-queue->lambda_func->lambda-dlq
That's true. The retry that actually worked for him seems to be from the lambda-queue configuration. Without adding lambda-dlq as a DLQ for lambda-queue, it would not have worked.
I thought the Lambda polls SQS for messages periodically. It should be asynchronous.
@@SchoolScience You are absolutely right. What is being displayed in the video is not wrong but, imo, misleading. The Lambda DLQ setup here is not needed. You can either do:
1) Queue to DLQ setup
This is essentially what the video shows. Create two queues, set one queue to be the DLQ, create a lambda trigger but DO NOT set a DLQ config in lambda. Post a message to the queue and check that it ends up in the DLQ
2) Lambda to DLQ setup
Create a queue and set the lambda DLQ config to be the queue you just created. Now invoke you lambda asynchronously (for example by creating another lambda that invokes your "DLQed" lambda with type "Event"). Notice that your "DLQed" lambda failed invocations now end up in your queue WITHOUT having to setup a second queue.
Hope this helps whoever is struggling with this as I was
Thank you Soumil😊
You’re welcome
Great Video Soumil, straight to the point. I have a question. I am trying to use SNS FIFO --> SQS FIFO --> lambda function for processing. SNS and SQS are Content-based deduplication enabled. If i pass one record working fine but if i send multiple producer messages, one record being processed rest are being deleting. Any help?
Can you explain more didn’t understand scenarios
@@SoumilShah thanks for the response. In simple, SQS FIFO triggers lambda function. SQS sent 4 messages but lambda invoked twice. Concurrency is "Unreserved account concurrency & 1000". What can go wrong?
@@anil1970 the reason lambda is invoked twice data is sent in batches in lambda lambda one got 2 records and lambda 2 got 2 records
Also depends on if you are using king or short polling and what batch size window you are using !
@@SoumilShah batch size was 10 and short polling. I changed batch size to 1. do you suggest anything more?
@@anil1970 what is your ask do you want 10 messages is that what you want ? Increas delay in sqs queue
Delay is time after which message appears in sqs
Hey Soumil, Did you ever do the serverless framework for this topic? Instead of UI, maybe CDK?
Yup there is video and lab I guess was lab 23 or 22 or 24 check out those I am sure I have one
I have a question, if I don't want my entire batch to get rejected in case there is an error in specific record. Can I push those particular records to the DLQ using boto3 calls instead of configuring DLQ on lambda with retey counts which will eventually rejects the entire batch or it is not a good design practice?
Yes but that would be issue
The term used for that is poison messages
Usually poison messages are better handelled in kinesis then dlq yes if you have try catch and if you just want to publish that message you can add that in catch blok )
Gotcha, yes have implement it inside the try-catch block. Thanks for the reply
@@shishir2697 anytime more than happy to talk more if you have Questions
great video
yes, please slow down yourself in every respect pls