The Pattern You MUST Learn in .NET
ฝัง
- เผยแพร่เมื่อ 30 ก.ย. 2024
- Use code MODULAR and get 20% off the brand new "Getting Started with Modular Monoliths" course on Dometrain: dometrain.com/...
Get the source code: mailchi.mp/dom...
Become a Patreon and get special perks: / nickchapsas
Hello, everybody, I'm Nick, and in this video, I will introduce you to the Outbox or Transactional Outbox Pattern in .NET. I will explain what the outbox pattern is and how it works, and we will see how we can use the MassTransit library to implement it.
This video is sponsored by AWS
Workshops: bit.ly/nickwor...
Don't forget to comment, like and subscribe :)
Social Media:
Follow me on GitHub: github.com/Elf...
Follow me on Twitter: / nickchapsas
Connect on LinkedIn: / nick-chapsas
Keep coding merch: keepcoding.shop
#csharp #dotnet
I want to know what is your other approach to doing this!
+1
The "listen to yourself pattern" is more scalable in my opinion
Another way, back in the day you could have another program you run and register a row change listener in SQL, you can receive an event any time a row in a table is added/change/delete. This registered call back could place the "outbox message" into the outbox table for example, and then i've see signalR used for pushing events to subscribers.
+1 I normally use cdc direct from DB transport events to message bus or queue, also good when you re happen to change the data directly on the DB you want miss the messages
@@ahupond Listen to yourself works, provided it does not cause timing issues with a consumer knowing about the message before the producer. If the Db supports it another trick is to listen to its transaction log, and write to your outbox from that. The issue there is that your code that knows how to create the outgoing message has to be able to interpret the outgoing message from the Db transaction. The reason why Outbox is so common is it doesn't bump into these issues.
"Hello everybody I'm naked" once you hear it you can't unhear it 😂
what have you done to me
Hellovrybody
Clowns 😂
This video is confusing since is not showing how to implement the design pattern, its showing how to use a black box that already has the design pattern.
Hey Nick, great video as always!
One thing was a bit unclear to me, which was the total message flow, so I did a little research.
From the video it seems that it is only one pattern you are using and that both services still need access to the same database. But in fact, you are using two patterns: the outbox and inbox pattern. The message is first saved to the outbox in the database. Then behind the scenes, MassTransit picks it up and sends it on the message queue (and removes it from the outbox table). Finally, the message is received by your worker which will save it to the inbox table. Then the worker reads from the inbox and processes the messages.
So the worker and API can be running as separate processes (on different nodes) and they can work with a different database. The worker only touches the inbox table and the API touches the outbox tables.
Correct me if I'm wrong, cause I only did a quick 5 minute research to clarify for myself :)
One question regarding the worker. Considering the behavior we're trying to solve is usually part of a microservice architecture, or some fully separated systems, doesn't the fact that the worker needs access to the outbox kind of defeat the purpose?
What I mean is that if I have access to the outbox, I have access to the DB. If I have access to the DB I can create a trigger for example in the DB, or a scheduled task that gets the info and handles it. I don't see a purpose of the queue in these cases, because I can just create an artificial one locally.
I think the idea is that you create another service that has access to both DB and MessageBroker. It reads data from the outbox and passes it into the queue. Other services consuming from the queue don't need connection to DB since they can use message acknowledgement to avoid duplicate processing of the message.
I think the example in the video was a bit confusing but that's not how it works. MassTransit automatically configures a Background Service in the API that polls the Outbox and sends the messages. The consumer application does not change, it still reads from the queue.
If you're using the database as the message queue/broker, you should be aware of SQL Hints, particularly UPDLOCK, HOLDLOCK, and READPAST.
UPDLOCK will make it take the necessary exclusive locks to modify the row,
HOLDLOCK will make it hold the locks until the end of the transaction (e.g. shared locks tend to get released otherwise),
READPAST makes it skip rows that are already locked.
Combine this with a SELECT TOP 1 read, and you've got yourself an easy and fairly robust (though not idempotent) queue.
Can run as many competing readers on it as you like (to diminishing returns), and they won't step on each other's toes.
That all said, I'm sure MassTransit has stuff built in to deal with any idempotency concerns.
I don't think the worker has access to the DB in the example
consumer reads messages from the queue (not db). It can use inbox table for duplicate detection feature and it can have it's own database.
I have heard about this pattern twice for past 2 weekends on interviews, great topic
yet another hyped nonsense as previuos ones....Fashion changes every few months..
Yes, I wan't to hear your preferred way of dealing with this problem.
The videos where you are introducing us to libraries and all that is great.
Outside of that, in my opinion, there is a consistent lack of disconnect between your intent and what you are really trying to accomplish through pattern-type videos.
You need explain a lot more technical background via system design and diagrams without even touching the code. Of course, this is not to a suggestion to make hour long videos. At least share system examples you have worked on and why in the hindsight you think you were right or wrong.
I do not really care you talking about food delivery service or ride-hailing service. That would be so because you have not worked on them.
Any heavily used enterprise system would pose massive technical challenges, so there should be enough to discuss on that itself.
If only seeing through the complexities of the system would be easier.
A common issue with the polling worker is that if you scale your app you would then have multiple workers processing and publishing the same messages. That's why people would usually process the outbox before acknowledging the original message. If the publish fail then that original message would retry, you would see that an outbox already exists and then only process the outbox instead of the entire message.
You can deal with it by adding a field like "ProcessingStartedAt" which will prevent messages from being processed by other workers until the one who picked them will complete processing and delete the messages. Having a date instead of bool will allow you to re-process messages which became stuck because of a crushed worker.
Hey Nick,
Could you please tell us your favorite way of solving this problem,
And also tell us about how we can guarantee message is handled only once in the consumer? Also wondering how to preserve correct order of the messages.😊
There is no need to configure the consumer side. Doing so only makes this example confusing (as people are already asking if it uses a database instead of a queue).
The consumer can be also configured to use an outbox pattern (which actually combines inbox and outbox), but it's not obligatory.
No... I Will not mix messaging with my application data layer.
It's a resiliency layer. The messaging is still handled by a messaging service.
@ I am not concerned about how or what handles the service but the implementation. If you watch the video, you will see that the implementation becomes part of your data access code. Nick briefly makes an excellent and concise explanation using a diagram. I prefer a separate service to handle messaging; it is not part of my application and deploys separately.
@@2SHARP4UIQ150 I see your point but there's probably not a good way to create a transaction that would affect both the data and the messaging service. The goal is to ensure that either both or neither of the writes happen. So yes, messages are temporarily (probably for less than a second) saved in the database along with the other data. It's a trade off but makes sense to me.
I want to know your favorite way of solving the problem!
Hi Nick, thank you for sharing. Please give us some more insight on your experience dealing with the alternative method for the outbox pattern.
Yes, please also show your approach to this problem.
Jeff Bezos: Hey Nick wanna have some money? Please make a video about some Amazon s***
Yes I’d really like to know your preferred solution to this problem. I’m not a big fan of this approach, it smells a bit “automagic” - i.e, it isn’t obvious when looking at the service code that the queue message isn’t published until the save changes transaction. I can imagine picking up this codebase and scratching my head as to why the message isn’t in the queue when it looks like it should be.
Instead of using entity framework in this case, I would manage the db in my code. I would start a transaction, stage the update, send message to the queue. If successful, I commit the transaction, if not, I rollback.
Hey Nick, I wan't to hear your preferred way of dealing with this problem.
Excellent video as always. I'm interested in hearing about your other approach that you mentioned at the end of the video. Cheers
I'd love to see the "Nick" approach to this, you tease 😊
it is worth noting the trade-off here. By assuming you are on EF Core, Mass Transit can avoid you needing to inform it about the "unit of work" that the Outbox needs to participate in. For Brighter we take a different view, and support multiple approaches to your DB: Dapper, EF Core, DynamoDb etc. but you need to pass us a unit of work. We will create this for you, via config, so that you can just use DI to inject it into your code. You then use the unit of work to create the transaction for your Db and the publish to the queue. As ever, trade-offs, but it removes the hard dependency on EF Core for us.
Yo dawg, I heard you like queues so we added a queue for adding messages to a queue so you can add message to a queue while adding message to a queue!
literally. I am looking and thinking, wth? The whole point of MQ was its simplicity of use, even replacing a MQ engine is not such a big deal that it reauires this massive MassTransit abstraction on top of it.
@@MilYanXo Yeah, the problems you run into using MassTransit for a while are not worth it in my opinion.
How does duplicate message detection work, particularly if the subscriber has many instances and does not have access to the DB?
The subscriber must have access to the database
TL;DR - Have your services store messages in a database table and then have separate service that sends them from there to your message broker.
Please let me know how you are dealing with distributed transactions.
Good video. I don't like the 'behind the scenes' aspect. Especially the invisibility of what is actually happening.
I'd be interested in know what you have actually implemented and used in practice for you previous requirements.
Atomic vs Self-healing is always a fun conversation. I've had good success with self-healing designs where atomic was not feasible. Could you do a video on self-healing design patterns?
Love you're "very scientific tool" as you mentionned :D
Why does the Worker also need the Outbox pattern and Entity Framework as a dependency? Please watch 15min.
Aren't you just shifting the problem?
Initially, you had the issue that you couldn't guarantee that both the database transaction was successful and that the message was placed on the queue. You have now solved that problem. However, there is now another, additional process that also needs to read a message from the database, place it on the queue, and then mark in the database that the message has been placed on the queue. Doesn't that process have the same problem? That it might place the message on the queue but fail to mark the message in the database?
Isn't it still an issue that you want to perform both a database transaction and a queue transaction?
12:32 I am confused about how id created and is passed to outbox table which was done before save changes.
Does the masstransist handles to update id and send to outbox table?
In EF you can implement interceptor for SaveChanges operation - so this is how MassTransit, most probably, does that
@@ealeykin got it thanks
Please create a video for your preference for dealing with this issue, thanks
What is publishEndpoint? This was not explained. It can't be any custom object.
I like to let you know that i am interested to learn how you solve that problem without a outbox.
Is it the build-in transaction-support in some messaging frameworks (i think azure service bus supports that).
I am so excited to read this comment
Interesting to hear about pettern for reverse situation. When you read from queue, and something happens with database, or the next queue sending
This example is terrible. But thanks for bringing up the topic, I'll dig into it by miself :)
I've been seeing this pattern a lot lately. Many teams implementing it at work. What is the other approach that you suggest (besides two-phase commit, obviously)?
how does mass transit know that the event shall be tied to customers table ? what will happen when there are multiple queue calls ? perhaps storing data into multiple tables etc.
It seems it deleted "physically" the event after processing. Can we soft delete instead?
But what happens if there is no consumer? Is there a timeout and api returns error?
Nick, with the pattern you ensure the message is published exactly once, but how can you make sure the message is also consumed exactly once?
It includes a "lock" column which should be enough for queries to ignore it if it's not filled with zeroes. I didn't see it but it must also have a "processed" column or something too.
The message queue should support locking so a message can only be consumed by a single subscriber.
He didn't explain this.
From what I read, one solution is to save message id's to be able to identify already handled message. If already handled - then skip.
Thanks once more, Nick! Very interesting pattern! I want to know about the other approach using native cloud products!
Nice! Hey what's that high-tech uml tool you using?
Thanks for covered topic, but for me seems like overkill. Interesting how everyone deal with this problem, from my side I don't see any problems with creating transaction, save changes, publish message , complete transaction. Interesting about other opinions.
What happens if after you publish the message but before you complete the transaction, any of the following happens: a network outage, app pool recycles, hardware failure occurs, or web process is shut down?
@@TheMikernet Got it, thanks,I understand the problem. To implement all this staff you need background worker which read DB (possible bottleneck) and push messages to queue. And background works can have the same problem (network, hardware) so we have to add some logic to prevent problem with background worker and logic to prevent duplication of messages in queue. Maybe it's better avoid using the outbox pattern and just add logic to subscriber of this event where we check that operation completed. Or if it is so critical to use Service Bus transaction. Thanks why I think that this pattern is overkill.
Anyone else not getting the email when registering in the dometrain course?
Awesome video on Information Systems. I think this was a great introduction to the development side of Information Systems. I think you touched on it, but did not say anything specifically, but what about Event Sourcing. And I vote yes on a video detailing your alternative method of solving this problem.
But does the worker has to connects to the same database?
The worker doesn't need to be aware of the outbox. There are multiple types of setup you can have and one of them is for the worker to be agnostic of the db stuff
thanks for replying@@nickchapsas, I really liked it the concept and started exploring it in a demo project.
I want to know what is your other approach to doing this!
13:52 Somebody knows what console it is?
At 16:37, you use AddAsync() to prepare adding a customer to a database. Why? Normally you would only need to use the Add() method unless you have some reason to use AddAsync()? The documentation says as much (e.g. SequenceHiLo).
Why there are three tables, and what is inbox?
Defo want to hear the other solution, currently building this very thing at work and we lol'd becuase your released the vid as soon as our meeting about it was done
I'm really spooked because this literally happened just after a meeting I had which spoke about the outbox pattern
Hey Nick,
I would like to congratulate you on another excellent video.
I have a question regarding to the need to add EF Core packages to Consumer.
The way I was assuming it work is following:
- publish push the event to the outbox table
- a background process (on the publish side) is responsible for scanning the table, pushing de event to the queue, and the update de status of the event row as processed
- the consumer still consumes messages from the queue with no changes needed
What am I missing here? 😅
7:30 When you said "Don't log like this", to me it was slightly unclear what you were referring to. Do you mean we shouldn't have log lines that change the semantic behavior of a function? Or do you mean abusing the message queue to log something that the creator of the message should have logged?
It's hard to tell exactly what you meant, and I can think of multiple reasons why it might not be a great idea.
He was referring to the concept called "Structured Logging"
He meant don't use the object .ToString() as a log message template. Why he didn't just do it "right" is beyond me though, would have taken less time.
Its so funny to see the resurrection of MSMQ spinoffs now that cloud providers decided to sell (err, rent) them. I've been there 3000 years ago when MSMQ with DTS was used for all this and we didn't call it outbox pattern :D
Hey Nick,
I would like to congratulate you on another excellent video.
I have a question regarding to the need to add EF Core packages to Consumer.
The way I was assuming it work is following:
- publish push the event to the outbox table
- a background process (on the publish side) is responsible for scanning the table, pushing de event to the queue, and the update de status of the event row as processed
- the consumer still consumes messages from the queue with no changes needed
What am I missing here? 😅
In this approach is there a behind the scenes polling of the outbox? If so, that could be costly from a cloud perspective. I've seen patterns that inert into a db, then add a service bus message with a reference to the record in the db, then ack to the client. You need to have a cleanup process if there is a failure between db and service bus, but there would be much less db polling. Not quite as elegant as the Listen To Yourself pattern, but it works with large messages that might not be suitable for service bus
Hey Nick, you mention the sponsored AWS course is free from the link in the description, but there is no link? And the course has a price tag on the website? Interested, thanks!
How would we configure something like this if we were trying to queue multiple transactions between multiple microservices, each one independent of each other... I wonder... 🤔
Love the abstraction over message queuing that MT offers. And this feature you've demonstrated is definitely something I'm keeping on the radar. Last project that had something similar was using MSSQL ServiceBroker, but that was ages ago and that technology has become a long forgotten relic. Very cool stuff, Nick.
Would you send the message to the queue AFTER saving the record to the database? What if you get a DB exception and the message was already propagated?
Does somebody use MassTransit with IBM MQ? Doesn't seem like it is supported :(
A good follow-up for this would be a video detailing the Inbox Pattern, for receiving messages.
I use outbox all the time. What's your preferred way to do it? We use masstransit, SQS, RMQ & rdb but not the built-in masstransit outbox.
Hi Nick, very nice video!
One question:
In api methods that work with related entities(aggregates), when for example I want to update all data in related in main and related entity, is a good idea to pass the related entity id or should get directly with the main entity id all the childs and update then?
Sorry, maybe the explanation is not the best...
The question is if should I send directly to the api related entity's IDs or should I obtain them from the main aggregate?
Thanks in advance.
Idk if I miss something about DI injection magics but who guarantees that Publish will work after inserting to database? Isn't the same problem as the beginning?
I believe they're both added to the changetracker which EF will ensure happens atomically (they either both are created or neither is). So it's not one then the other. It's either both or neither.
The Publish method is hijacked behind the scenes and it turns into a database operation that is part of the transaction
Thanks! @@nickchapsas
The very idea that there's a line of code that says Publish and yet nothing seems get published after it's execution bothers me so much, there's so much going on that is invisible to me that it seems like I would never use it.
one of the things I learned is that not everything has to be programmed. There are many automation tools that can be spawned in a docker container to do this kind of background processing. In this case you just set a deleted flag on your record, then your cron job will spin up and do all the extra processing like email or clean up without having to rewrite all that code yourself and bloat it.
When you need to write the message to the DB - why not simply pull the DB via an IP or push some notification to a consumer of the "mesage" service that sometging int he DB has changed
So far I used Azure Functions but this looks interesting, I also wanted to know what other approach do you suggest, thanks for the great turor
One of many things I came up with myself and then got to know it exists as a common practice. Ughh
We have a legacy system that implements something like the outbox system. Some action will write a message to our custom outbox and some separate background app will check that outbox and send the message. But it doesn't have retries and no idempotency.
Love videos like this - explain useful pattern and show a good way to implement it using OSS 🙌
Too bad that there is no easy way to stop listening for messages from the queue. Or maybe there is? What if my app received some message that starts up some heavy process and I don't want to listen on new messages like this from the queue until I'm ready?
Is there already a video out which is Nick's favourite approach to this ?
I'm working on it
@@nickchapsas Thanks. Blessings to you in your work!
"ντάξει"
The outbox pattern is applicable for some use cases, in your example it is not the case though. You added extra complexity to your design and pooling in the database. A DTC with retries would simpler, atomic and efficient.
Glad im not the only one using paint to explian things! Brother id love to see a video on the other approach you mentioned to solve for atomic operations over a database and queue!
I too would like to know the alternative approach, especially if it's cloud centric.
Why does the working need to use DBContext? didn't it just listen to sqs and process?
@Nick Chapsas - I would like to know your approach, please.
a very scientific tool...opens up paint. Great content man. Many tanks
I write to a table have a status column and my service reads from it :/
use function app with cosmos db trigger, cosmos db is your outbox
Why can't you please tag API videos? A simple "#api" (or better yet, #webapi) would help people who are looking for API videos (SEO FTW), and give realistic expectations for desktop developers. I mean, I'll continue watching the video, but I definitely don't "need" to learn the pattern. Probably.
Because it’s not API specific. Any kind of service can publish to a queue
This is nice. I solve things like this just with a Task / Thread Process in the background and a messagequeue as a command/message store to process dependend tasks like this. It is simple and easy and only as many features in this construct that i need for my problem that i have to solve. But i really love this content, because it validates my way of doing things, if big companies walk same ways in a bigger radius of problem solving.
Another fantastic video, definately interested in the other option.
I want to know what is your other approach to doing this!
I think I like the simplistic of save changes the fire the message. Having a relay poll the database and then enqueue the message seems complicated and oddly familiar to using the database as a queue. Maybe my problems aren’t enterprise-enough to warrant this.
But in enterprise scale of micro services each service has its own database and common queue. So it is not the same as DB will be pulled only by one service just to publish messages to the queue while in your solution all services would need to access single DB and pull it.
Pub sub is the best. I demo this in paint for our new hires.
Could someone clarify why there is a service that reads from the outbox table and pushes the message to a queue to be read and dealt with as opposed to just resding from the outbox table and dealing with the messages at that point in time?
IE read and process directly from the outbox table and cut out the message queue.
Other microservices probably have its own DBs. Still if you have a single DB for all then you need to pull it even more often because each service will be pulling it. And you loose built in support for topics etc for queues.
@Nick, is it possible to use with LiteDB?
Is there a reason you use insomnia now instead of postman?
I wanna know your preferred approach very much
The only issue I can see consumer should have access to publisher database or outbox configuration should be enough on publisher side only?
It doesn't need to have access. It depends on your setup
It would be great to hear how else you deal with this.
I'm also curious about the other approach
Why use await to publish to the queue?