Want to get more hands on with Lambda? Check out my course, AWS Lambda - A Practical Guide - www.udemy.com/course/aws-lambda-a-practical-guide/?referralCode=F6D1A50467E579C65372
From the title of the video I could tell exactly what the problem was going to be. This is one of the weird things about scaling with SQS that does catch a lot of people out and it's not well documented at all. Really easy to fix once you understand what's going on though
Great video. For a moment I thought you were about to talk about the fact your SQS queue must have VisibleMessages available in order to the Lambda to decide scaling up. I have a Lambda with a weird behaviour in which it NEVER fails the execution (by design) but I still don’t see it scaling. My SQS queue is always showing VisibleMessages as 0 and I don’t know why that happen. Not sure if you have seen anything like that.
Another weird thing: I wasted 16 hours. Last week my prod went down and I found out the ssl listener has been removed from CLB. I added them again and five minutes later same thing. After a lot of digging found out there is no more support for CLB anymore, the migrate button didn’t do anything and AWS has never communicated this with us. Super Frustrating!!
In the end when you mentioned people were removing messages from sqs which are successfully processed. When i was working with sqs as lambda trigger, i saw that all the sqs messages received by lambda were automatically deleted irrespective of they get processed by lambd aor not. Same thing i saw on many stackover flow answers
The issue is when you are using batch sqs processing, you receive many messages at once per invocation. If you successfully process say 9 out of the 10, you need to ensure the 9 get deleted and the one failure remains.
If you get 10 messages sent by sqs and successfully process 8 of them, will you return a list of those 2 ids in the lambda response? And maybe you have a sample of that lambda? Thank you.
Hi. So you mean when throw error with try catch. Or just simply throw “something with no catch” ? Is it considered error when return statusCode:400 ? I just try to experiment lambda/sns/cloudwatch week ago. 😂
no you have to return something like this { batchItemFailures: [ { itemIdentifier: messageId }, ] } and make sure you configure event source mapping and add your sqs uuid to it
Want to get more hands on with Lambda? Check out my course, AWS Lambda - A Practical Guide - www.udemy.com/course/aws-lambda-a-practical-guide/?referralCode=F6D1A50467E579C65372
I had this issue and I didn't even know, now it's fixed thanks to you. Great content!!
Glad it helped!
Really good information. Thanks for sharing
..."what the hell is happening here?" 😆literally my mindset everyday in dev
There is rarely a day where I don't mutter this to myself haha.
Great info on both handling of lambda concurrency and error handling…Thanks for sharing !
Love this. I have learned so much about AWS thank to you, thank you so much man !!!
From the title of the video I could tell exactly what the problem was going to be. This is one of the weird things about scaling with SQS that does catch a lot of people out and it's not well documented at all. Really easy to fix once you understand what's going on though
It definitely is a less known quirk of SQS and Lambda scaling. I’m glad I’m not the only one that thinks the documentation is lacking :p
That was a good find. Amazing solution by Lambda on this. Thanks for sharing mate. Thumbs Up!!
Glad you enjoyed !
Excelent video!!! very helpfully
That's really interesting. My big takeaway is partial batch responses, I forgot about those, but it really seems like everyone should be using them
Absolutely! I never realized this could become a problem until it did :P
That is a very good feature it is used to apply back pressure, it limits function invocation automatically, I was looking for this
Thank you!
Can you provide an example of the partial batch error in javascript?
Great video. For a moment I thought you were about to talk about the fact your SQS queue must have VisibleMessages available in order to the Lambda to decide scaling up. I have a Lambda with a weird behaviour in which it NEVER fails the execution (by design) but I still don’t see it scaling. My SQS queue is always showing VisibleMessages as 0 and I don’t know why that happen. Not sure if you have seen anything like that.
I just switch to EventBridge & EventBridge Pipes in my current project, I am surprised by its capabilities and ease of use
great lectures, great teacher.
That’s a helpful tip. Thanks.
Another weird thing: I wasted 16 hours. Last week my prod went down and I found out the ssl listener has been removed from CLB. I added them again and five minutes later same thing. After a lot of digging found out there is no more support for CLB anymore, the migrate button didn’t do anything and AWS has never communicated this with us. Super Frustrating!!
In the end when you mentioned people were removing messages from sqs which are successfully processed.
When i was working with sqs as lambda trigger, i saw that all the sqs messages received by lambda were automatically deleted irrespective of they get processed by lambd aor not.
Same thing i saw on many stackover flow answers
The issue is when you are using batch sqs processing, you receive many messages at once per invocation. If you successfully process say 9 out of the 10, you need to ensure the 9 get deleted and the one failure remains.
@@BeABetterDev okay
If you get 10 messages sent by sqs and successfully process 8 of them, will you return a list of those 2 ids in the lambda response? And maybe you have a sample of that lambda? Thank you.
Really cool thx
@BeABetterDev, does this apply for batch size of 1?
Hi. So you mean when throw error with try catch. Or just simply throw “something with no catch” ? Is it considered error when return statusCode:400 ? I just try to experiment lambda/sns/cloudwatch week ago. 😂
no you have to return something like this
{
batchItemFailures: [
{
itemIdentifier: messageId
},
]
}
and make sure you configure event source mapping and add your sqs uuid to it
Is it possible for you show the demo becuase many of them can not visualize the tech infprmation ❤
learned a lot
This is gold
Was fixed in December 2023 by setting max concurrency param
Any chance you could update your Udemy course with this issue and its fix ? It could help a lot of students. Thanks!
check the docs, everyone checks the docs they walk you through the steps
🙏👍