Thank for the video and I really enjoy your Microservices 101 series, Wade! Quick question, what type of system would you typically store the event log in? My assumption would before this video be systems like Apache Kafka, but your drawings in this video suggest that there is a separate storage system for the event log?
Awesome video, thanks for doing this. Question: what strategy would you use to share the event log with other microservices? It seems like you have to copy everything from one service database to another microservice database. Kafka has retention period which is not suitable for event sourcing, I guess? This question follows me everytime I hear about event sourcing. How can we sync event log records among microservices in a correct way?
I'm still trying to understand the database architecture for this. Will there be a single 'EventLog' table to store events for all aggregates, with the other tables only used for projection purposes?
A video on progress tracking problems would be great. Let's say event A triggers various events in different microservices. How can a microservice determine when all events that were produced in response to event A are fully processed? For example, if event A is a file upload, which triggers virus scanning, format conversion, and metadata extraction, how can we track when all related processes are complete? Some of those processes are optional or depend on context, and the total amount of processing to be done can be unpredictable. Additionally, there may be no sequential order for these processes. I also find progress hard to track if you can't use historic data and don't know the total amount of work that needs to be done. Is coupling and increasing complexity inevitable in such cases? Let's say one microservice consumes a FileProcessingQueued event which triggers a not easily predictable amount of other events, and another microservice expects a FileProcessingFinished event, while some other microservice may expect progress reports even before FileProcessingFinished is produced. This problem (or anti-pattern) deserves its own name and strategies for dealing with it.
@@ConfluentDeveloperRelations I believe many devs may find it valuable. After more learning, I can see it may be related to choreography and orchestration. Having progress reporting or completion events in choreography seems impossible. Much like trying to predict Conway's Game of Life
It's really great but it comes with a big cost, and it is the checkpointing implementation complexity. Because our audit logs grow over time, there should be a stateful mechanism to memories the last event read, otherwise each time an application restarts, it has to read all events at once. Using an associated framework to deploy event-sourcing is better than implementing it ourselves.
When an application restarts it brings all relevant individual domains into its memory one by one. For an application that has dozens of Domains, it can be not valuable to separate the queries per each domain, because eventually all of the events need to be queried. A solution for using streaming or paging to query events is fine, but remember that those old events should be processed, and at the sometime new events will be generated as a result. A solution to scale it is using CQRS is viable however considering that the events are stored in one table, scaling a database might be easy but scaling a table is not, therefore here we just helped with separating read and write but adding new challenge which is the inconsistency issues. Implementing an event-streaming tool, is not the business of many companies. My experience with working in event-streaming showed that using a dedicated event-streaming platform is beneficial, but it also can be blocking if our core business relies on it. All of these challenges are because of database being as our source of truth. I'd like to discuss this more of possibility of using Kafka as our event-sourcing. Kafka store events, we can replay event, we can scale easily, we can monitor events, we can use full-data-compatibility features out of the box. It would be great if you make a video about this in future.
@@ConfluentDeveloperRelations partially agree with your points. Unfortunately, I've witnessed instances where developers have tampered with events in database after a domain field updated for example. While developers can edit event logs in a database, they cannot do so in Kafka. Also, I noticed that since database is earlier than Kafka people who are expertise in DDD design, tend to stick to database as a familiar tool that they trust. Over decade Kafka supports more than just being a simple event-streaming platform, transactions supported, compact topics, locking is supported, there was time we could say that Kafka is not enough. Some argue that Kafka can't guarantee ordering of messages as it's crucial in global event ordering, However, with careful design and configuration, Kafka can still achieve a high degree of event ordering and consistency. Also, my point when I said, we have database as a one point of failure, is because it can't scale as good as Kafka does. Even using CQRS will create another eventual consistency to our system. We need to implement a busy pull mechanism from projectors, and this put a lot of pressure on database - remember the first initial idea to have CQRS was to lift the pressure from database. So, IMHO while we can use database for implementing a nitty gritty event sourcing, we can evaluate Kafka that already provide them.
Having a traditional database with two hundred tables, 400GB size, running complex joins and relying on fast retrieval of current state... I just cannot imagine how we could query a virtually endless eventlog to compute the current state... What is the bridge to go from a relational database to this sexy eventlog?
Mostly because in event sourcing, you validate before, based in the current events, to ensure the business rules apply. So you can never get in an inconsistent state.
@@aflous might be, but event sourcing just needs the events to be stored somewhere, they don't need to be on a log. It's just all very much tied to the capabilities of Kafka, while Kafka is far from ideal for event sourcing. That's why event sourcing with Kafka is hard.
![เปิดใจ "อาจารย์เบียร์ คนตื่นธรรม" อดีตเคยสุดทุกทาง ถ้าใช้หนี้หมดไปบวชแน่ l [Nickynachat]](http://i.ytimg.com/vi/IxeE0xVhy08/mqdefault.jpg)
This was clearly explained. Thank you very much.
Thank for the video and I really enjoy your Microservices 101 series, Wade! Quick question, what type of system would you typically store the event log in? My assumption would before this video be systems like Apache Kafka, but your drawings in this video suggest that there is a separate storage system for the event log?
Nice explanation in simple video about Event Sourcing Pattern.
Awesome video, thanks for doing this.
Question: what strategy would you use to share the event log with other microservices? It seems like you have to copy everything from one service database to another microservice database. Kafka has retention period which is not suitable for event sourcing, I guess? This question follows me everytime I hear about event sourcing. How can we sync event log records among microservices in a correct way?
I'm still trying to understand the database architecture for this. Will there be a single 'EventLog' table to store events for all aggregates, with the other tables only used for projection purposes?
A video on progress tracking problems would be great. Let's say event A triggers various events in different microservices. How can a microservice determine when all events that were produced in response to event A are fully processed?
For example, if event A is a file upload, which triggers virus scanning, format conversion, and metadata extraction, how can we track when all related processes are complete? Some of those processes are optional or depend on context, and the total amount of processing to be done can be unpredictable. Additionally, there may be no sequential order for these processes. I also find progress hard to track if you can't use historic data and don't know the total amount of work that needs to be done. Is coupling and increasing complexity inevitable in such cases?
Let's say one microservice consumes a FileProcessingQueued event which triggers a not easily predictable amount of other events, and another microservice expects a FileProcessingFinished event, while some other microservice may expect progress reports even before FileProcessingFinished is produced.
This problem (or anti-pattern) deserves its own name and strategies for dealing with it.
@@ConfluentDeveloperRelations I believe many devs may find it valuable. After more learning, I can see it may be related to choreography and orchestration. Having progress reporting or completion events in choreography seems impossible. Much like trying to predict Conway's Game of Life
Great video and clear explanation
this was very nice explained
It's really great but it comes with a big cost, and it is the checkpointing implementation complexity. Because our audit logs grow over time, there should be a stateful mechanism to memories the last event read, otherwise each time an application restarts, it has to read all events at once. Using an associated framework to deploy event-sourcing is better than implementing it ourselves.
@@ConfluentDeveloperRelations Hi Wade, thanks for your valuable response.
When an application restarts it brings all relevant individual domains into its memory one by one. For an application that has dozens of Domains, it can be not valuable to separate the queries per each domain, because eventually all of the events need to be queried.
A solution for using streaming or paging to query events is fine, but remember that those old events should be processed, and at the sometime new events will be generated as a result. A solution to scale it is using CQRS is viable however considering that the events are stored in one table, scaling a database might be easy but scaling a table is not, therefore here we just helped with separating read and write but adding new challenge which is the inconsistency issues.
Implementing an event-streaming tool, is not the business of many companies.
My experience with working in event-streaming showed that using a dedicated event-streaming platform is beneficial, but it also can be blocking if our core business relies on it.
All of these challenges are because of database being as our source of truth.
I'd like to discuss this more of possibility of using Kafka as our event-sourcing. Kafka store events, we can replay event, we can scale easily, we can monitor events, we can use full-data-compatibility features out of the box. It would be great if you make a video about this in future.
@@ConfluentDeveloperRelations partially agree with your points.
Unfortunately, I've witnessed instances where developers have tampered with events in database after a domain field updated for example.
While developers can edit event logs in a database, they cannot do so in Kafka.
Also, I noticed that since database is earlier than Kafka people who are expertise in DDD design, tend to stick to database as a familiar tool that they trust. Over decade Kafka supports more than just being a simple event-streaming platform, transactions supported, compact topics, locking is supported, there was time we could say that Kafka is not enough.
Some argue that Kafka can't guarantee ordering of messages as it's crucial in global event ordering, However, with careful design and configuration, Kafka can still achieve a high degree of event ordering and consistency.
Also, my point when I said, we have database as a one point of failure, is because it can't scale as good as Kafka does. Even using CQRS will create another eventual consistency to our system. We need to implement a busy pull mechanism from projectors, and this put a lot of pressure on database - remember the first initial idea to have CQRS was to lift the pressure from database.
So, IMHO while we can use database for implementing a nitty gritty event sourcing, we can evaluate Kafka that already provide them.
Having a traditional database with two hundred tables, 400GB size, running complex joins and relying on fast retrieval of current state... I just cannot imagine how we could query a virtually endless eventlog to compute the current state... What is the bridge to go from a relational database to this sexy eventlog?
Very informative.
what is sourcing event managment ?
Great stuff
Thank god for Temporal
hamilitonian vs langrangian!
Wrong, on many levels.
How come?
Mostly because in event sourcing, you validate before, based in the current events, to ensure the business rules apply. So you can never get in an inconsistent state.
@@ConfluentDeveloperRelations so the video is not about event sourcing?
@@gerardklijs9277you're not getting the point here
@@aflous might be, but event sourcing just needs the events to be stored somewhere, they don't need to be on a log. It's just all very much tied to the capabilities of Kafka, while Kafka is far from ideal for event sourcing. That's why event sourcing with Kafka is hard.