Getting started with Queues in .NET using AWS SQS
ฝัง
- เผยแพร่เมื่อ 6 ก.ค. 2024
- Get $50 free AWS Credit by adding "AWS CREDIT #NICK" in the "Tell us about your project" area: bit.ly/nickaws
Get the source code: github.com/Elfocrash/aws-videos
Check out my courses: dometrain.com
Become a Patreon and get source code access: / nickchapsas
This video is sponsored by AWS
Hello everybody I'm Nick and in this video I will show you how you can get started with Queues in C# and .NET by using the AWS SQS service. Queues are a crucial part of any microservice architecture system and SQS is one of the most popular AWS services. Knowing how to use it is a critical part of any software engineer's skillset.
Don't forget to comment, like and subscribe :)
Timestamps
Intro & creating the SQS Queue - 0:00
Creating the Publisher - 5:29
Creating the Consumer - 14:59
Extending the consumer with handlers - 23:00
Automatic registration refactoring - 35:38
Video on queues: • What is Azure Service ...
Social Media:
Follow me on GitHub: bit.ly/ChapsasGitHub
Follow me on Twitter: bit.ly/ChapsasTwitter
Connect on LinkedIn: bit.ly/ChapsasLinkedIn
Keep coding merch: keepcoding.shop
#aws #dotnet #sqs
This is the best C# channel on TH-cam x 10. Sometimes it's like Nick has access to my backlog and makes videos just for me.
A simpler way to have it working nice is with Mediatr, the background service does de parse and dispatch of the messages
I would like to see the covariance and contravariance one!
Your channel is amazing. It's perfectly in-depth enough to lean and start doing. As requested, more AWS + .NET Architecture videos please.
This is honestly my favorite video. So many good patterns and examples. Hats off.
Can't wait for the architecture video ! Keep up the good work !
I like the fact that your videos and Topics are getting "bigger" and more detailed! thx
Nick your videos are just what is required, a true scientist... well done man.
Yesssssssss, been waiting for this, I've done a few implementations in my projects but I've never been fully happy with it. Would love to see more videos on proper async data pipelines/flows👍 thanks for doing these videos man, I dropped out in first year of my Cs degree and while I can code, I always feel lacking on the infrastructure/architecture side. Your courses and videos have been a massive help for me🙏
Hey Nick, great video! Thank you bringing such quality content!
Great info... having used MSMQ for over 10 years for Process Control.. nice to see a modern implementation of a message dispatcher
I mean even 10 years ago MSMQ wasn't "modern". Wondering why it was ever picked after 2008.
Really nice video! Congrats! And as always, keep them coming… 🎉
Eyy, this was super helpful and very easy to follow. Thank you! 🙏
Congratulations Nick, as always a good video.
I would love to see different techniques how to tackle idempotency! I've seen many different approaches and would be great to compare with what you've seen.
The message received by SQS has one property messageId and that id is unique no metter how many times this message is delivered, a simple way to deal with this idempotency is saving this Id
Cool video! I’d probably prefer to use a message bus like mass transit to abstract most of this out for me, but still fun to see it built too
As usual, awesome content. 🙏
I love how you're releasing these AWS videos a few months after I had to figure it all out myself lol
Hey Nick, awesome video, thanks so much for the great content and for sharing. Would be interested in how to deal with idempotency too. Looking forward to the solution architecture video too.
Yo Nick! I'm Xamarin dev that switched to backend half a year ago. Old viewer of yours, but now - your channel is a real GEM for me, every new video I watch is exactly what I need, just started my AWS journey and the timing of this one is magic. Thank you for your content! Also I really want to see a bonkers approach with Handlers and more AWS content overall.
I've been using this approach for processing messages via the azure service bus since generic host (background service) released in .net core 2.1. Thanks for producing these videos, I find the content very useful and practical.
Thanks for sharing all those details. I am also using SQS very intensively and love the service for its simplicity. Keep it up
So this is a fascinating idea.
The way you described Standard AWS QUeue sounds like a prioritized asynchronous queue.
I can imagine a queue where all of these items are waiting to be processed, and as product becomes available it can be filled by the first order that asks for it, maybe.
But its potentially optimizable because if you have someone in slot 3 asking for grapes, but slot 2 wants grapes and pomegranates but the pomegranets are delayed (and maybe you optimize to try to fill what you can complete, and if it ends up getting stuck then getting intervention somehow to break the queued item up. AWS could optimize to look for the first item in the queue that can be filled and dispatched. So I think I really like this idea, because its not like you have competing tasks waiting to be filled and potential contention that way. Pretty cool!
Hi Nick, great video as always. Is there any chance you could do a video on Amazon Cognito? i.e. basic setup and walkthrough
Hey Nick, great video, please do more about queues and how to efficiently publish/consume messages 🙏.
Kind of unrelated, but it would make sense to put together handler's registration and populating MessageDispatcher's dictionaries? (to avoid forgetting registering them in Program.cs)
Many thanks!
No need for dictionaries, just use something like Scrutor to automatically register stuff
I'd love to see a video for the generic message handling. That scenario comes up way too often.
Great video and pretty slick use of reflection to handle registering your services in the service container. Do you have a video that would utilize SNS and also display how the messages are deleted from the Queue? Also, would be nice to see the use of the DLQ for failed messages.
Awesome video, although the last part 'Automatic registration refactoring' is a little bit hard and rush.
I tried to run this project in dotnet 7 but it fails because IMessageHandler can be used as a type. Then I downgrade to dotnet 6 but I can use abstract in the property IMessageHandler.MessageType so I have to initialize that property with typeof(IMessageHandler), it is good ?
Can one use NServicebus with SQS and not worry about implementing a custom dispatcher logic?
Hi Nick. Maybe in the near future you could create video about SAGAs?
Looks like you ended up not using the static abstract interface members feature as the reflection scanning would work regardless. Right?
Hey Nick. Thanks for the video. I have a question though. If the published message is subscribed by multiple applications (or I can say users), how do we make sure all the users see the message before it gets deleted? (If we don't delete the message, one user might end up seeing the message multiple times)
Hey Nick, you don't have a link in the description below for Queues.
Sorry added now
Your comment about the messages being in a shared project instead of duplicating the files is a good one. However when you used the executing assembly it will not find those messages, instead typeof(IMessage).Assembly would be better. Same concept for the IMessageHandler.
Yeah I did mention that you would use a type or an assembly if you had it in a separate project
@@nickchapsas Now that I think about it more, like you said most developers would have understood to use typeof(IMessage) instead of typeof(Program) if they moved the files to a separate project.
Great video, reduce font size please
Hi Nick! Do you use Azure (in this case the service bus) as well and can you make a comparison, or are you only on AWS?
I’ve used both in high scale production environments for 3 years each and I prefer AWS.
@@nickchapsas interesting, i would love to see a video about the reasons some day :)
Very nice video. Any chance of making a similar vid with MassTransit?
I plan to make an expansion on this video to show how MassTransit can make all this better
Great vid bro... can you do some on Dapr?
Amazing content and great explanations. But for a more pleasant reception, I would advise 10% slower. It isn't a chase :).
Did you just write your own MediatR? =)
heh, I decided to implement MessageDispatcher with generics, and it was the worse decision for the last few days) Could you please make a video\another branch in github with implementation in the way with generics/variance stuff.
I’m planning to use this example in a future video to explain the runtime problems (which you probably encountered), variance and alternatives
@@nickchapsas That would be awesome. Thanks
sadly we cant use aws services in iran,thats just make me feel depressed
For the dispatcher, I'd create an IEnumerable
IMessageHandler can contains
{
string Command {get => nameof(MessageType)}
Type MessageType {get;}
Task HandleAsync ();
}
Then register all handlers as IMessageHandler
Finally inject IEnumerable in the constructor
Using linq
var handlers = msgHandlers.Where(mh => mh.Command == input.commans )
foreach(var handler in handlers) await handler.HandleAsync(input.Data);
This approach is problematic because it instantiates all handlers to just use one. waste of speed and memory
Make sense, that's why you created a dictionary
Unnecessarily instantiating handlers aside, i like this approach. I would probably introduce an abstract base class like this:
public abstract class MessageHandler : IMessageHandler where TMessage : IMessage
{
public async Task HandleAsync(IMessage message)
{
if (messageis TMessage m)
{
await DoHandleAsync(m);
}
}
protected abstract Task DoHandleAsync(TMessage m);
}
so my MessageDispatcher is nice a simple like this:
public class MessageDispatcher
{
private readonly IServiceScopeFactory scopeFactory;
public MessageDispatcher(IServiceScopeFactory scopeFactory)
{
this.scopeFactory = scopeFactory;
}
public async Task Dispatch(IMessage m)
{
using var scope = scopeFactory.CreateScope();
foreach (var handler in scope.ServiceProvider.GetServices())
{
await handler.HandleAsync(m);
}
}
}
This way you can also write handlers which work off a base message type for instance:
public class AllMessagesHandler : MessageHandler
{
protected override Task DoHandleAsync(IMessage m)
{
Console.WriteLine(m.GetType());
return Task.CompletedTask;
}
}
Hey mate, your videos are so useful, can you please make a video on comparison between Entity Framework latest vs Dapper. Why we should use dapper over entity framework and use cases.
👏🏽👏🏽👏🏽👏🏽👏🏽
@Nick Chapas, How can i rate limit a specific api which url contains and require int value.
E.g
api/account/details - This is like you explained - which is working
but what in case if url is like this?
api/account/{accountId:int}/details
Please help me. im using webapi2
5:13 Am I only one person who expected a random id as 69
I never understand the obsession with adding `Async` to methods where you don't provide a sync version of it. It's just fluff. Everyone knows a task can be awaited
Then in the code that you write you should definitely not use that suffix
@@nickchapsas I definitely don’t. In fact calling it Async may be a lie. See MemoryStream for example. I also don’t add access modifiers just because; only when necessary.
@@ShiyalaKohny To clarify, I never meant to be rude and if that’s how it came across, I want you to know it wasn’t the case @Nick Chapsas.
@@paulkoopmans4620 Microsoft don’t even follow their own conventions on their asp code base 😂. Async is just noise as is typing private for private members.
@@metaltyphoonnot sure which occurrences you are thinking of at this moment. But... no framework and/or programming is ever 100% correct. That does not mean however that "standards" and trying to follow them is bad. Quite the opposite IMHO.
I will admit that I come back to my code at times thinking... wt... why did I just not follow the coding style guide or why I not do this or that. But I am a human and so are they.
But if nothing would have existed at all.... oh boy... I can just imagine how terrible that must be. Property names will all kinds of different casing. Terrible method names like b_convstr() instead of TryConvertString(), just because, you know. No!!! No!!! No!!! Even though never perfect, millions of developers following a "standard" is better than nothing.
And again; I personally believe in intent of code. If a member is a private member it should be marked as such. If it were up to me I would say the implicit access modifier behaviour should have never existed. It serves no purpose! the "choice" of private as the default access modifier is as terrible of a choice as any of the other ones.
My classes are usually sprinkled with lots of public methods, public properties, protected, etc, as well. I think I may usually have more of any of the other ones than private. Maybe public should have been the default :).
god forbid any poor soul has to write such C#
Yeah, get on AWS, so they can turn you off if you have the "wrong ideas". No thanks.
While I agree with you, I believe that regular developers don't need to worry about it as much as our clients - they're the ones that amazon might refuse service. We can only advise them not to use services from providers who might be unreliable like that.
@@marcinpawelw The problem arises when it's your own AWS-backed service that a client is using. They can wrongthink your entire service in to the ground. Not a liability I want to take on
I like your videos but you speak so fast. Could you speak more slowly, please?