If you want a primer on what Event Sourcing is check out this video and a playlist talking about Event Sourcing, Projections, Snapshots, and more: th-cam.com/video/AUj4M-st3ic/w-d-xo.html
Thank you. You are amazingly clear in your explainations! No useless fillers and anecdotes, love that you just get right into the nitty gritty while everyone else seems to get lost in the nutter butter.
Great video! I would love to see in more detail how you implement the projection side. Especially when adding a new projection to an existing system and having to replay events for current state
Hi, quick question 🙋 I wonder if CQRS needs to be separate the service into two standalone service. Or it can be sit within the same service but having two “handlers”? Cause then we don’t need to face the eventually consistent problem
Event Sourcing and CQRS aren’t the same thing but almost always come as a package deal. ES is optimised for writes (no locking contention) so you’re always going to end up considering a specific, separate design for the reading side. So in reality you don’t do ES without CQRS. That’s why people see them as different sides to the same coin.
Thanks for the video! One question popped up in my mind. At around 12:15 in your video you say that you are fetching the item from the event store by applying all of the events. Why do you need to do this? - that seems like a waste of time (and potentially a very expensive thing to do, if there were many events for the item). At this point, you don't use the item for anything - you just store the "count increment" event and trigger the projections.
You want to build up current state to enforce invariants. It's only expensive if the event stream is really long, which generally isn't. If it is and designed correctly you can optimize by using snapshots. th-cam.com/video/eAIkomEid1Y/w-d-xo.html
You are the best out there! I just have one question. How do you ensure both databases are always sinchronized and no event was lost in the network? Thanks for your videos Derek!
Good Question. Keep track of the version of the stream in your projections. Your event stream will have a version which is can be used for optimistic concurrency. Include the version in the event so when you're can check your projection to make sure you haven't missed any messages. Also, if you're using an actual database, you'd leverage something like subscriptions. As an example, check out how EventStoreDB does it: developers.eventstore.com/clients/dotnet/5.0/subscriptions.html#volatile-subscriptions
Hi, thanks for very instructive videos. I have a couple question about this video. Sorry, if I'm missing some basics even I've checked your other videos. So, It seems that total count of an item is calculated in Projections Handler. Doesn't it mean business logic leaks out from the Aggregate Root? And in case of a new projection which requires total count, this logic would be duplicated, right?
I wouldn't view it as leaking business logic. You aren't enforcing any invariants in projections, simply using domain understanding to create the count based on the events. If a new projection required the total count, it could also contain that logic, yes. But nothing stopping it from being shared. Eg, having a projection that is purely for "stock count" that other queries or projections use.
@@CodeOpinion Ok, I understand; although the projection and the actual state of total count may diverge at any point of time, this won't break the application since all the validation and the mutations is being done by the aggregate root. However, this doesn't prevent the chance of an inconsistency. For instance, if i change the calculation logic of total count and I forget to apply this change in projection, data would be inconsistent. So, how about if I pass the whole aggregate root or just the total count invariant to the ItemsCheckedInToInventory event? Eventually, I have the current state at that point.
Can u pls implement with kafka, what do u think? And also video how to have consistency between reads n writes with segregated dbs how would u r update read DB after writes in write DB??
On your diagram 2:18 event for querydb fires and in parallel is saving to eventstore db? Does it make sense? What if saving to eventstore db fails - we will have wrong projection in querydb. Maybe better re-emit event for querydb after save to eventstore db (eventstoredb subscription)?
No you likely don't want to be saving to your event store then trying to publish. This would require a distributed transaction/2PC. I mention this later in the video here: th-cam.com/video/5aznkIEvkKc/w-d-xo.html
Hi Derek. First of all amazing video as always. Second i wanted to ask a question. How would you manage in this case the event replay for reconstructing the projection database. As i understood from the code base, you recreate the current state of the aggregate root every time you read it from the event store, however since the events replayed are marked as not new, no new event is published for the handlers to capture. My question is: if I wanted to recreate the projection database from the current events I already have, how would I do it in this architecture. Thanks as always!
To be clear, in an aggregate root, you're replaying all the events from a stream to get to current state within the aggregate root, which ultimately you use against business logic. For projections, the intent is to always have a durable/persisted representation of the most recent event. When a new event is published, you consume that event and update your projection. So basically you're projection is constantly being kept into a current state as new events are created/published. If for some reason you want to rebuild a new or existing projection, you would request all the events from whatever streams you need to rebuild. If you haven't already, check out my post on projections: th-cam.com/video/bTRjO6JK4Ws/w-d-xo.html
@@CodeOpinion Thanks for the answer, however I am afraid I did not phrase the question correctly. I was referring to the read model projection. In this case the read model is only being updated in real time and it is synced with the latest state. However how would I manage the event reply in the case when I want to generate a new read model projection or maybe fix a corruption in the projection. In the video you mentioned the problem is easier since it is an in-memory projection, however how would we manage a persistent one. Thanks and apologies for the long question.
To rebuild from scratch, you'd start with a blank read model them start reading the stream from index 0 and build up the projection again. This is often called a catch up subscription because the client keeps track of where in the stream it is. Once caught up you can move to a persistent subscription to be fed where you stopped and get pushed all the latest events as they happen.
Im new to .NET core for web and not sure What stack should I use for simple Website with 3D model display and preview from database (Best Way for 3d Model store database not sure which one to pick). Maximum 50-100 models in my db.
I don't get why there are 2 tables for the same entity. Should the database really have 2 tables for the same entity for presentation purposes ? Doesn't seem right
One question: which should be the response from the REST api to the UI after a command to create an entity is received? 200 OK 201 CREATED 202 ACCEPTED
Love how u explain complex topics. I had a question. In the context of event sourcing whenever there is fault the solution is to replay back to the working state and analyze what happened. if u are using state machine or orchestration design, does that model eliminate the need to replay due to the fact that if there is fault, the state machine will not transition to the next state, thus preventing the need to look back in the event store. What are our thoughts?
If you persist an event to your event stream due to a bug, it happened, there's no turning back. At that point you'd have to handle that when replaying the event stream to get to current state, even if that puts you in to an invalid state. Likely what you would have is another compensating event to "fix" the error. You'll likely create an event that will "undo" or "void" the incorrect event.
@@CodeOpinion oh that's right. I forgot that I was actually doing a compensating transaction whenever the failed event is persisted to correct the issue. Thanks
Hi, trying to implement CQRS+ES and i have a question: What are common approaches to handle async between write and read on client side? After sending a command how to notify client about projection update completion to run query safely? In my case i trying to develop "smart" WS gateway to implement "subscribing on changes with query run", but it will be nice to hear some best practices, thx.
It really depends on the context. You can use SignalR (websockets) or some type of push to the client. But it really depends on the context and where you're applying event sourcing. Users often expect to immediately see their changes, in a given context, so don't fight that. That may be something that's CRUD in a nature and you're event sourcing when you shouldn't.
@@CodeOpinion That's weak point of DDD + CQRS + ES popularization. Of cause if you dive deep into this you should know, that everything depends of context, but when you build software production flow with teams you want to give them some templates, or at least some generic abstract technology approaches. Clean architecture helps a lot, it gives bounds to put context-depended things, and devs are happy, but async write-read frontend - that`s is the problem that i can`t beat easily (
@@evgenvb This isn't just an issue with ES and Projections, this happens anytime you do any work asynchronously and you want to let the client know. Check out: th-cam.com/video/Tu1GEIhkIqU/w-d-xo.html
Thanks for this useful video, I have a question, here you save read side data after saving data in your event store in same process (with advantage of transaction). What happens if we have multiple read side technologies like mongo and elastic? Here we can't use same in process transaction for handling failure on both data store. I know eventstore-db has a projection feature for this case (async subscriptions). But how could we handle it ourselves?
Yes, likely you'll updated your projections (read models) asynchronously. Meaning your projections will be eventually consistent with the event stream. Check out my video on ways to handle eventual consistency: th-cam.com/video/wEUTMuRSZT0/w-d-xo.html
Great video. Question, do you think CQRS / MediatR pattern makes sense using a single DB? I'm working on a prj right and have the liberty to implement Clean Architecture & CQRS. However, hosting (azure cloud) comes with it's costs (like any other cloud provider). Hosting / provisioning 2 DBs is more $$$ than 1. Plus all the extra local storage / complexity with Docker to run locally and run tests during builds. I'm thinking a single CosmosDB, applying the same principles explained here, but without projections and event sourcing. Just a different workflow for commands & 1 for queries, most likely use separate repositories all together. Make sense or wasting my time here 🤣? Sorry for the long post Love your content btw
Great work thanks Derek. In case i have a form to update a person informations i wish track only modified informations how to do that in ES in term of events ?
in terms of use, are you able to describe, when it's best to use event sourcing and when inbox/outbox pattern is enough? I can see similar concept here and inbox/outbox pattern could also be used to republish events.
Inbox/Outbox server different purposes than Event Sourcing. The Outbox is to solve the issue of not having a distributed transaction (2PC) with your db and message broker. The inbox is because of "at least once" you need to make your consumers idempotent as you may process the same message more than once. Event Sourcing is about how you store data, not about messaging (although you can).
What does the InventoryItemReository and why are you creating a new InventoryItem entity? Isn’t the whole point of Event Sourcing to avoid the Entity Based storage as a source of truth? For it looks like you’re storing your usual Entity via the repository, and then notify the write models. Shouldn’t you instead of this store the creation event?
The InventoryItem is the aggregate root. The constructor generates an InventoryItemCreated event. The repository isn't persisting the aggregate but the events it's creating.
@@CodeOpinion So while handling of a command, an event can be produced then applied to the state, then another event can be produced and applied to the state. As a result of processing one command zero or more events can be produced and applied to the state. Right?
Why writes are called commands and Not writes. A command is Any action if you speak plain English? Sp query is also command. Just say write and read fo what they are
Writes implies changing state. Eg, making a write to a database. However a command is just performing some type of of action with a side-effect. Sending an email is a command,
If you want a primer on what Event Sourcing is check out this video and a playlist talking about Event Sourcing, Projections, Snapshots, and more: th-cam.com/video/AUj4M-st3ic/w-d-xo.html
Thank you. You are amazingly clear in your explainations! No useless fillers and anecdotes, love that you just get right into the nitty gritty while everyone else seems to get lost in the nutter butter.
Thanks!
You’re my hero! This is the exact thing I’ve been looking for. Like the final piece of the puzzle in my mind! Thank you!
No problem 👍
The only problem with this channel is that it's not that popular as it deserves to be.
Great video! I would love to see in more detail how you implement the projection side. Especially when adding a new projection to an existing system and having to replay events for current state
Great suggestion!
Always great stuff! When I get a notification, I'm always eager to check out a new event sourcing video from you. Thanks!
Awesome, thank you!
So the projection mainly code updates the read database in turn of a business event?Thnks!
Yes the projection is updating the read DB.
Hi, quick question 🙋 I wonder if CQRS needs to be separate the service into two standalone service. Or it can be sit within the same service but having two “handlers”? Cause then we don’t need to face the eventually consistent problem
Event Sourcing and CQRS aren’t the same thing but almost always come as a package deal. ES is optimised for writes (no locking contention) so you’re always going to end up considering a specific, separate design for the reading side. So in reality you don’t do ES without CQRS. That’s why people see them as different sides to the same coin.
Hey @Derek, you're doing amazing. Thank you! Please add corresponding github repo link with all of your videos. Thanks Again.
nice, I'm waiting for the source code to follow along
It's been added. Check out the community tab! Thanks again for your support!
Thanks for the video!
One question popped up in my mind. At around 12:15 in your video you say that you are fetching the item from the event store by applying all of the events. Why do you need to do this? - that seems like a waste of time (and potentially a very expensive thing to do, if there were many events for the item). At this point, you don't use the item for anything - you just store the "count increment" event and trigger the projections.
You want to build up current state to enforce invariants. It's only expensive if the event stream is really long, which generally isn't. If it is and designed correctly you can optimize by using snapshots. th-cam.com/video/eAIkomEid1Y/w-d-xo.html
Thanks! Just to confirm if your command api needs to read the current state of the entity before insertion then youd still need to replay events ?
Yes, you're getting all the events and replaying them to build up current state so you can apply any invariants (business logic)
You are the best out there! I just have one question. How do you ensure both databases are always sinchronized and no event was lost in the network?
Thanks for your videos Derek!
Good Question. Keep track of the version of the stream in your projections. Your event stream will have a version which is can be used for optimistic concurrency. Include the version in the event so when you're can check your projection to make sure you haven't missed any messages. Also, if you're using an actual database, you'd leverage something like subscriptions. As an example, check out how EventStoreDB does it: developers.eventstore.com/clients/dotnet/5.0/subscriptions.html#volatile-subscriptions
Hi, thanks for very instructive videos. I have a couple question about this video. Sorry, if I'm missing some basics even I've checked your other videos. So, It seems that total count of an item is calculated in Projections Handler. Doesn't it mean business logic leaks out from the Aggregate Root? And in case of a new projection which requires total count, this logic would be duplicated, right?
I wouldn't view it as leaking business logic. You aren't enforcing any invariants in projections, simply using domain understanding to create the count based on the events. If a new projection required the total count, it could also contain that logic, yes. But nothing stopping it from being shared. Eg, having a projection that is purely for "stock count" that other queries or projections use.
@@CodeOpinion Ok, I understand; although the projection and the actual state of total count may diverge at any point of time, this won't break the application since all the validation and the mutations is being done by the aggregate root. However, this doesn't prevent the chance of an inconsistency. For instance, if i change the calculation logic of total count and I forget to apply this change in projection, data would be inconsistent. So, how about if I pass the whole aggregate root or just the total count invariant to the ItemsCheckedInToInventory event? Eventually, I have the current state at that point.
好视频，有图有代码看着很清晰，感谢。
Can u pls implement with kafka, what do u think? And also video how to have consistency between reads n writes with segregated dbs how would u r update read DB after writes in write DB??
GOOD explaining man , thank you
You're welcome!
Again such a fantastic video. Thank you :)
Glad you enjoyed it!
On your diagram 2:18 event for querydb fires and in parallel is saving to eventstore db? Does it make sense? What if saving to eventstore db fails - we will have wrong projection in querydb.
Maybe better re-emit event for querydb after save to eventstore db (eventstoredb subscription)?
No you likely don't want to be saving to your event store then trying to publish. This would require a distributed transaction/2PC. I mention this later in the video here: th-cam.com/video/5aznkIEvkKc/w-d-xo.html
Hi Derek. First of all amazing video as always. Second i wanted to ask a question. How would you manage in this case the event replay for reconstructing the projection database. As i understood from the code base, you recreate the current state of the aggregate root every time you read it from the event store, however since the events replayed are marked as not new, no new event is published for the handlers to capture. My question is: if I wanted to recreate the projection database from the current events I already have, how would I do it in this architecture. Thanks as always!
To be clear, in an aggregate root, you're replaying all the events from a stream to get to current state within the aggregate root, which ultimately you use against business logic. For projections, the intent is to always have a durable/persisted representation of the most recent event. When a new event is published, you consume that event and update your projection. So basically you're projection is constantly being kept into a current state as new events are created/published. If for some reason you want to rebuild a new or existing projection, you would request all the events from whatever streams you need to rebuild. If you haven't already, check out my post on projections: th-cam.com/video/bTRjO6JK4Ws/w-d-xo.html
@@CodeOpinion Thanks for the answer, however I am afraid I did not phrase the question correctly. I was referring to the read model projection. In this case the read model is only being updated in real time and it is synced with the latest state. However how would I manage the event reply in the case when I want to generate a new read model projection or maybe fix a corruption in the projection. In the video you mentioned the problem is easier since it is an in-memory projection, however how would we manage a persistent one. Thanks and apologies for the long question.
To rebuild from scratch, you'd start with a blank read model them start reading the stream from index 0 and build up the projection again. This is often called a catch up subscription because the client keeps track of where in the stream it is. Once caught up you can move to a persistent subscription to be fed where you stopped and get pushed all the latest events as they happen.
@@CodeOpinion Amazing. Thank you for your help. Keep up with your amazing content.
Im new to .NET core for web and not sure What stack should I use for simple Website with 3D model display and preview from database (Best Way for 3d Model store database not sure which one to pick). Maximum 50-100 models in my db.
I don't get why there are 2 tables for the same entity. Should the database really have 2 tables for the same entity for presentation purposes ? Doesn't seem right
One question: which should be the response from the REST api to the UI after a command to create an entity is received?
200 OK
201 CREATED
202 ACCEPTED
Love how u explain complex topics. I had a question. In the context of event sourcing whenever there is fault the solution is to replay back to the working state and analyze what happened. if u are using state machine or orchestration design, does that model eliminate the need to replay due to the fact that if there is fault, the state machine will not transition to the next state, thus preventing the need to look back in the event store. What are our thoughts?
If you persist an event to your event stream due to a bug, it happened, there's no turning back. At that point you'd have to handle that when replaying the event stream to get to current state, even if that puts you in to an invalid state. Likely what you would have is another compensating event to "fix" the error. You'll likely create an event that will "undo" or "void" the incorrect event.
@@CodeOpinion oh that's right. I forgot that I was actually doing a compensating transaction whenever the failed event is persisted to correct the issue. Thanks
Hi, trying to implement CQRS+ES and i have a question:
What are common approaches to handle async between write and read on client side?
After sending a command how to notify client about projection update completion to run query safely?
In my case i trying to develop "smart" WS gateway to implement "subscribing on changes with query run", but it will be nice to hear some best practices, thx.
It really depends on the context. You can use SignalR (websockets) or some type of push to the client. But it really depends on the context and where you're applying event sourcing. Users often expect to immediately see their changes, in a given context, so don't fight that. That may be something that's CRUD in a nature and you're event sourcing when you shouldn't.
@@CodeOpinion That's weak point of DDD + CQRS + ES popularization. Of cause if you dive deep into this you should know, that everything depends of context, but when you build software production flow with teams you want to give them some templates, or at least some generic abstract technology approaches. Clean architecture helps a lot, it gives bounds to put context-depended things, and devs are happy, but async write-read frontend - that`s is the problem that i can`t beat easily (
@@evgenvb This isn't just an issue with ES and Projections, this happens anytime you do any work asynchronously and you want to let the client know. Check out: th-cam.com/video/Tu1GEIhkIqU/w-d-xo.html
Thanks for this useful video,
I have a question, here you save read side data after saving data in your event store in same process (with advantage of transaction).
What happens if we have multiple read side technologies like mongo and elastic?
Here we can't use same in process transaction for handling failure on both data store. I know eventstore-db has a projection feature for this case (async subscriptions).
But how could we handle it ourselves?
Yes, likely you'll updated your projections (read models) asynchronously. Meaning your projections will be eventually consistent with the event stream. Check out my video on ways to handle eventual consistency: th-cam.com/video/wEUTMuRSZT0/w-d-xo.html
@@CodeOpinion Thanks
Great video. Question, do you think CQRS / MediatR pattern makes sense using a single DB? I'm working on a prj right and have the liberty to implement Clean Architecture & CQRS. However, hosting (azure cloud) comes with it's costs (like any other cloud provider). Hosting / provisioning 2 DBs is more $$$ than 1. Plus all the extra local storage / complexity with Docker to run locally and run tests during builds.
I'm thinking a single CosmosDB, applying the same principles explained here, but without projections and event sourcing. Just a different workflow for commands & 1 for queries, most likely use separate repositories all together.
Make sense or wasting my time here 🤣? Sorry for the long post
Love your content btw
Yes, CQRS isn't about having different databases. It's about separating the path between reads and writes. The ultimately can go to the same database.
Great work thanks Derek.
In case i have a form to update a person informations i wish track only modified informations how to do that in ES in term of events ?
I found what i was looking for you've already answered the question in this th-cam.com/video/6-quY0PJUP4/w-d-xo.html
Also check out this video about creating Task Based UIs: th-cam.com/video/DjZepWrAKzM/w-d-xo.html
@@CodeOpinion Thank you
Thank for the walk-thru by the way can CQRS and ES be implemented on framework like Laravel?
I don't use PHP but there's really no need for a framework to apply CQRS or ES really.
@@CodeOpinion Appreciate it a lot. keep posting vids. it really helps for someone like me who just new to implementation of this architecture. 😉
in terms of use, are you able to describe, when it's best to use event sourcing and when inbox/outbox pattern is enough? I can see similar concept here and inbox/outbox pattern could also be used to republish events.
Inbox/Outbox server different purposes than Event Sourcing. The Outbox is to solve the issue of not having a distributed transaction (2PC) with your db and message broker. The inbox is because of "at least once" you need to make your consumers idempotent as you may process the same message more than once. Event Sourcing is about how you store data, not about messaging (although you can).
what database do you recommend as eventstore?
EventStoreDB, which is built for Event Sourcing. www.eventstore.com/
Note, they do sponsor my channel but that doesn't affect me referring them.
What does the InventoryItemReository and why are you creating a new InventoryItem entity? Isn’t the whole point of Event Sourcing to avoid the Entity Based storage as a source of truth? For it looks like you’re storing your usual Entity via the repository, and then notify the write models. Shouldn’t you instead of this store the creation event?
The InventoryItem is the aggregate root. The constructor generates an InventoryItemCreated event. The repository isn't persisting the aggregate but the events it's creating.
in minute 3 diagram, should the event be written first on the Event Store, then execute the event handler for projection. Not splitting it.
MVP vs MVP... lol
Yes. It's written to the event store first. You'd use a subscriptions to update the projections.
Does a command handling produce only one event?
No, it can produce more than one.
@@CodeOpinion So while handling of a command, an event can be produced then applied to the state, then another event can be produced and applied to the state. As a result of processing one command zero or more events can be produced and applied to the state. Right?
Why writes are called commands and Not writes. A command is Any action if you speak plain English? Sp query is also command. Just say write and read fo what they are
Writes implies changing state. Eg, making a write to a database. However a command is just performing some type of of action with a side-effect. Sending an email is a command,
@@CodeOpinion that is not true. We are talking about saving data and reading it. Bo one is interested in sending email here
Show me a case of cqrs that never writes and sends email and reads
no, I did not like asvu missed replay