Your explanations are straight to the point with very clear examples. I ended up watching the series from Getting the Basics to Event Sourcing and CQRS. Thanks for sharing your knowledge.
I've watched about 4-5 of this series so far and it's great! I love how succinct and clear you make the main points and the final presentations. Great job!
Diagram is more intuitive and easier to understand, so thank you for this. My question is: Imagine designing a backend system for a todo list application where data consistency across Redis, PostgreSQL, and Elasticsearch is essential. To achieve this while eliminating Dual Write, two distinct approaches can be considered: Kafka with Separate Topics and Consumers: Utilizing Kafka, this approach involves creating three separate topics ("todo-cache-ingestion", "todo-storage-ingestion", and "todo-search-ingestion"). If create API call to create a new todo would publish messages to these topics. Subsequently, distinct consumer processes would handle data storage in Redis, PostgreSQL, and Elasticsearch, respectively. and what is we fetch all or specific data here? and second approach is Event Sourcing approach? But How? and what is the issue in first approach?
Hello! Glad you liked the video! So probably event driven architecture doesn't fit the todo list case as well as other approaches, but let's use it for the example. One thing to consider here is that you can't have strong consistency in both reads and writes here. You may achieve eventual consistency. So for the fist case, if you read all the to dos from the list you need to decide from which system. Depending on the number of writes they may not be aligned (you may be able to search but not retrieve from cache for example). If there are no writes you may be able to eventually get them all in sync. First and second approach are similar. I'm not sure about the question. What did you mean?
Would event sourcing be appropriate for this eccommerce checkout example? I would imagine that when a customer is checking out they would like an immediate response as to whether their order/payment was successful. With this asynchronous model there's not a guarantee of this
Those are the pros and cons of event sourcing. Payment could be synchronous if needed. Some systems value the speed of giving the service to the user and then send an email if the payment doesn't go through (for example, Google play periodic payments work like this). But it all goes down to what do you need in your case
Hi. Really good series as I'm starting to learn about Event-Driven architecture. I couldn't differentiate Saga-Choreography from CQRS. Ideally in Choreography we would send commands/events from one service to others. And those messages can be a get/put. In CQRS aren't we are doing the same thing ? Like sending a message for get/query and sending a message for post/command. I'm jumbled up with the terminologies and everything sounds similar to me. Sorry If I'm not making any sense.
Hey Madhanraj don't worry! Learning can be hard sometimes but it's always worth! There are several things here let me try to explain it. First, design patterns / architecture patterns depend a lot on the intention. Depending on the intention certain details make sense. If you look at the details but don't see the big picture sometimes is harder to see / understand the pattern. For example you mention doing get/put to store information: get/put can be done on any service to device communication or for example between layers in a layered architecture, so it helps to see the intention and the bigger pattern. Now on the CQRS vs Choreography these are very different. In Choreography you are trying to make sure different services are coordinated by sending messages. You wouldn't do a "Get" you only send messages and services react to them. On CQRS the intention is to split the put from the get, sometimes in different services/modules/parts. That's it. You can implement it as you prefer. For example, you can have choreography between different service when you do a put,and then do a get directly from the storage or calling a other service. Hope it's clearer!:)
I want to ask about the Event Sourcing and CRUD. For example, an Authentication service has API to register new users and a News Feed service generates recommendation for that new user. In this case, Auth service can send USER_CREATED to the Event Log to follow the event sourcing. However, in the case that Event Log is not reachable then you will need an outbox pattern where the Auth service stores events in its local database and send to event log later. I want to ask if it is normal to not use CRUD in the register API and the register API will publish an REGISTER_USER event without storing the data, the auth service has an internal worker that will subscribe to the REGISTER_USER event to store data to its local database and then publish an USER_CREATED event for external services. In this case, no need to do an outbox pattern with the assumption that event log will be high availability.
The event log should always be high availability to prevent that pattern you are mentioning, but it can still happen. It's also OK to combine the patterns as you are mentioning too. At the end these are pieces that can be put together to solve a business problem in the best way possible. Here the typical principles like "Keep It Simple" are relevant. If you can solve the problem with CRUD solve it with it as it will be simpler. If you have multiple that need to be orchestrated and Event Driven works better for you, take that path. If you are having a huge disparity between reads and writes, Event Sourcing and CQRS can help too. Hope it's clearer :)
@@ADevStory thanks. I have better understanding now. The pattern where the api handlers don't perform write but publish and listen to the same events is called listen to yourself. However, for the user registration I believe that it is better to use CRUD directly at the handler and using outbox pattern to prevent event log downtime. The reason is that user's personal data is sensitive and it shouldn't be stored in the immutable event log.
It can still be stored in a secured event log. Just like the database needs to be secured. You can also have retention policy for the data in the log too
@@ADevStory upload vieos on system design .. like how aws works.. how gmail works .. like that ... explain architecture of every large scale systems ..
Hi can I mix a crud application with event sourcing? For example when I perfom an action into a Microservice A with http crud publish the event and microservice B make changes in a local cache?
Why on earth would the orders overview service query three different services instead of building its own internal state based on events from the other services? Seems like it would nullify the pros of using this pattern.
That's a good question. Default answer: there are many ways to build the same functionality, and your approach is valid, it all depends on the context. More detailed answer: if you have the order overview service recreate the full state from all the events directly you could get more coupling and less domain separation between the services. For example, if the events used change then you need to update your Overview service, and you need to pay attention for all the possible events that are used by the different services you query (Payment, Inventory, Shipping). Some could be even vendor specific events or integrated apis (imagine Shipping is reading DHL, UPS, and USPS events, while Payment is reading Mastercard, Visa, Paypal and Stripe events). If one of those events change, then you need to change it wherever they are used. Events also become apis. In this case (for educational purposes, your mileage may vary), by having the events organized by domains and having services covering those domains, it makes it simpler for the Order Overview service to just pull the already processed data from the other services, hence the usefulness of the CQRS pattern. Hope is clearer :)
Thank you. Your EDA series helped me pass in an interview.
Really happy to hear that!
Your explanations are straight to the point with very clear examples. I ended up watching the series from Getting the Basics to Event Sourcing and CQRS. Thanks for sharing your knowledge.
Glad you enjoyed it. I'm planning the next ones, any topics you would like to suggest?
You are a gifted Teacher. You sync between explanations and clean book diagram, just MindBlowing.
Thank you very much! Glad you enjoyed it!
I've watched about 4-5 of this series so far and it's great! I love how succinct and clear you make the main points and the final presentations. Great job!
Thank you very much! Glad it was useful! Please let me know of there are more topics you may be interested
Hello, thank you for your tutorial, can you make a tutorial axon framework with CQRS & SAGA PATTERN ?
I typically don't make videos specifics to frameworks but what exactly do you want to know? I also have a video o the saga pattern
I really much agree that at the end you said to limit the use of this pattern. Can be overkill most of the time
Exactly!
Increíble el contenido que hay en esta serie de 3 videos, muy resumida y clara la información. Saludos desde México!!
¡Qué bueno que te hayas gustado! ¡Saludos!
Super clear and simple explanations, thanks a lot for this series.
Glad you like them!
Diagram is more intuitive and easier to understand, so thank you for this.
My question is: Imagine designing a backend system for a todo list application where data consistency across Redis, PostgreSQL, and Elasticsearch is essential. To achieve this while eliminating Dual Write, two distinct approaches can be considered:
Kafka with Separate Topics and Consumers:
Utilizing Kafka, this approach involves creating three separate topics ("todo-cache-ingestion", "todo-storage-ingestion", and "todo-search-ingestion"). If create API call to create a new todo would publish messages to these topics. Subsequently, distinct consumer processes would handle data storage in Redis, PostgreSQL, and Elasticsearch, respectively. and what is we fetch all or specific data here?
and second approach is Event Sourcing approach? But How? and what is the issue in first approach?
Hello! Glad you liked the video!
So probably event driven architecture doesn't fit the todo list case as well as other approaches, but let's use it for the example.
One thing to consider here is that you can't have strong consistency in both reads and writes here. You may achieve eventual consistency.
So for the fist case, if you read all the to dos from the list you need to decide from which system. Depending on the number of writes they may not be aligned (you may be able to search but not retrieve from cache for example). If there are no writes you may be able to eventually get them all in sync.
First and second approach are similar. I'm not sure about the question. What did you mean?
Thank you for excellent explanations and keeping simple.
Thank you! Cheers!
Cool videos! I would recommend you to buy a microphone as sometimes it hard to understand because of eco. Great work, can't wait for the next video!
Thanks for the tip! I have mic and that has reduced the reverberance/echo, but will try with other mics in the future.
Thank you very much, these explanations are awesome!!
Thank you! Glad you enjoyed it!
Great described. Thanks!
You're welcome!
Would event sourcing be appropriate for this eccommerce checkout example? I would imagine that when a customer is checking out they would like an immediate response as to whether their order/payment was successful.
With this asynchronous model there's not a guarantee of this
Those are the pros and cons of event sourcing. Payment could be synchronous if needed. Some systems value the speed of giving the service to the user and then send an email if the payment doesn't go through (for example, Google play periodic payments work like this). But it all goes down to what do you need in your case
good way of explanation😄🙂
Thanks! Glad you liked it!
Hi. Really good series as I'm starting to learn about Event-Driven architecture. I couldn't differentiate Saga-Choreography from CQRS. Ideally in Choreography we would send commands/events from one service to others. And those messages can be a get/put. In CQRS aren't we are doing the same thing ? Like sending a message for get/query and sending a message for post/command. I'm jumbled up with the terminologies and everything sounds similar to me. Sorry If I'm not making any sense.
Hey Madhanraj don't worry! Learning can be hard sometimes but it's always worth!
There are several things here let me try to explain it. First, design patterns / architecture patterns depend a lot on the intention. Depending on the intention certain details make sense. If you look at the details but don't see the big picture sometimes is harder to see / understand the pattern. For example you mention doing get/put to store information: get/put can be done on any service to device communication or for example between layers in a layered architecture, so it helps to see the intention and the bigger pattern.
Now on the CQRS vs Choreography these are very different. In Choreography you are trying to make sure different services are coordinated by sending messages. You wouldn't do a "Get" you only send messages and services react to them.
On CQRS the intention is to split the put from the get, sometimes in different services/modules/parts. That's it. You can implement it as you prefer. For example, you can have choreography between different service when you do a put,and then do a get directly from the storage or calling a other service.
Hope it's clearer!:)
@@ADevStory Thanks for the detailed response. Much appreciated
Anytime!
nice explanation
Thanks!
I want to ask about the Event Sourcing and CRUD. For example, an Authentication service has API to register new users and a News Feed service generates recommendation for that new user.
In this case, Auth service can send USER_CREATED to the Event Log to follow the event sourcing. However, in the case that Event Log is not reachable then you will need an outbox pattern where the Auth service stores events in its local database and send to event log later.
I want to ask if it is normal to not use CRUD in the register API and the register API will publish an REGISTER_USER event without storing the data, the auth service has an internal worker that will subscribe to the REGISTER_USER event to store data to its local database and then publish an USER_CREATED event for external services. In this case, no need to do an outbox pattern with the assumption that event log will be high availability.
The event log should always be high availability to prevent that pattern you are mentioning, but it can still happen. It's also OK to combine the patterns as you are mentioning too. At the end these are pieces that can be put together to solve a business problem in the best way possible. Here the typical principles like "Keep It Simple" are relevant.
If you can solve the problem with CRUD solve it with it as it will be simpler. If you have multiple that need to be orchestrated and Event Driven works better for you, take that path. If you are having a huge disparity between reads and writes, Event Sourcing and CQRS can help too.
Hope it's clearer :)
@@ADevStory thanks. I have better understanding now. The pattern where the api handlers don't perform write but publish and listen to the same events is called listen to yourself. However, for the user registration I believe that it is better to use CRUD directly at the handler and using outbox pattern to prevent event log downtime. The reason is that user's personal data is sensitive and it shouldn't be stored in the immutable event log.
It can still be stored in a secured event log. Just like the database needs to be secured. You can also have retention policy for the data in the log too
I read on many blog posts that CQRS is not recommended for CRUD, now I am confused 😐
That's what I said. Not recommended for CRUD. If you can solve something with CRUD then skip CQRS. Hope it wasn't confusing
muchas gracias eres lo mejor
Oh gracias :)
keep uploading ..
Will try! Time is scarce these days 😅 what would you like to watch?
@@ADevStory upload vieos on system design .. like how aws works.. how gmail works .. like that ... explain architecture of every large scale systems ..
Sure!
@@ADevStory You mentioned "Time is scarce", how can I help, I would love to be your apprentice
@@IsaacLarbiOnline thanks! Not much really, mostly sharing the channel and video if you've found it interesting and suggest topics :)
Hi can I mix a crud application with event sourcing? For example when I perfom an action into a Microservice A with http crud publish the event and microservice B make changes in a local cache?
Yes of course! So service B is doing event sourcing (and potentially CQRS) but service A is doing just a regular CRUD. That's perfectly fine.
Why on earth would the orders overview service query three different services instead of building its own internal state based on events from the other services? Seems like it would nullify the pros of using this pattern.
That's a good question. Default answer: there are many ways to build the same functionality, and your approach is valid, it all depends on the context.
More detailed answer: if you have the order overview service recreate the full state from all the events directly you could get more coupling and less domain separation between the services. For example, if the events used change then you need to update your Overview service, and you need to pay attention for all the possible events that are used by the different services you query (Payment, Inventory, Shipping). Some could be even vendor specific events or integrated apis (imagine Shipping is reading DHL, UPS, and USPS events, while Payment is reading Mastercard, Visa, Paypal and Stripe events). If one of those events change, then you need to change it wherever they are used. Events also become apis.
In this case (for educational purposes, your mileage may vary), by having the events organized by domains and having services covering those domains, it makes it simpler for the Order Overview service to just pull the already processed data from the other services, hence the usefulness of the CQRS pattern.
Hope is clearer :)
Hi guys, Can I call EventLog EventStore? Because I read many post and they mentioned EventStore.
Yes it can be used interchangeably
Shouldn’t you have two different data stores?
Not necessarily. Or even not desirable. The pattern is more about the access to the datastore than the datastore itself
I think it makes more sense to have two different data stores. For example you may not need many indexes on write and several indexes in read
CQRS not well explained: not enough details
Sorry it wasn't useful to you. What would you have liked I added?