the title is soooo confusing!! The cover photo says "Event Sourcing" and the title says "Event Driven". I believe they are quite different things. Now you helped me mixed them up, thank you :)
So not exactly, his architecture diagram depicts a service that captures the events in an event store as it's dispatched on the bus which is event sourcing. Event driven architecture version, is just the process of using events to communicate between services via event bus whether internally or externally and have each service subscribing to that event to later process it. What he hasn't covered in depth was replaying events, but he briefly mentioned stream processing when he mentioned rollback when he gave examples and advantages. Guys this is an overview video not an in-depth one. I believe he did a great job. Good man keep it up!
Its pretty cool of you to put this out! I might not pick up everything the first time, but its really elluminating and I really think about trying out new tech... thanks for that.
These videos are really good. More better, your excitement explaining things. There are lot of videos in youtube which may be better than your but man they are boring like hell that I get sleepy. Keep up the good work!
To explain event-driven architecture.... Would be helpful to not add in another layer on top in the example one is providing. That's what event sourcing does.
I believe why people like your video's is the way you enjoy the concepts and complete involve and explain them.. we can see in your eyes ( joy of happiness ) ... It could be great if you can show a video to community .. system design discussion with in group of people .. like everyone present their idea and how will you figure out the best in that room ...
Hi Gaurav, First of all, thank you so much for the content which you have highlighted on the "Event-Driver Architecture". One of the disadvantages which you have highlighted was "unable to track the end-2-end flow" in this architecture. So to answer that we can use the "Orchestration" pattern and you might be aware. So, basically, design the "workflow-orchestration" as soon as you get the service response so that we'll come to know where it hangs in case of any failure. We're using "CAMUNDA" for the same. Thanks!
Thanks Gaurav. NIce video. May I also point out a few advantages of event driven architectures? 1. It is the only way in which you can implement SEDA and hence scale individual stages as load increases. As the EDA works in eventual consistency model, the scale up times are not as relevant (or as relevant as they may be in req/resp paradigm) 2. This is necessary if you are implementing a CQRS based system. 3. Lambda architecture is only supported with events (allowing us to arguably beat the CAPs theoren) 4. In container world, where a container can be evicted any time (in the middle of a transaction etc), we either need all services to be idempotent or have an event driven architecture in place. Idempotency is not always possible. Regards
I think this is more of event sourcing architecture (which is sub category of event driven) than event driven itself. In event driven architecture, my understanding is that there is no "replay" function nor there is implementation to support the replay functionality. A good example of it is almost all front end frameworks use event driven archs and they don't store any state
great video thanks so much. you could talk at length, endlessly, about event driven architecture, and this video actually was the perfect level of depth. event driven architecture, or subsets of it, are absolutely used in just about every single tech company these days. however, one of the hardest parts is figuring out what happens downstream after a message is published. can require major logging tools on top of entire systems like splunk or ELK to trace what happens through your systems, which is complex on it's own. i've spent days or weeks tracing transactions through microservice queues trying to solve bugs. Also, enforcing message schemas is another huge problem. rabbitMQ / kafka + nodeJS is a particularly nasty combination when it comes to debugging LOL. but i suppose thats why we get paid the big bucks ;)
Hey Gaurav! Your videos are great. Every topic has been explained to the perfect depth without making it boring. Just a minor concern I have, can you please order your videos correctly like sometimes a video that's coming up next is something that I don't know anything about which I know you have explained in some other video but don't know which one. It'll be great if you do that! Otherwise everything's great!
Gaurav, Multiple observations 1. You seemed to have confused event log with local db for each service. The micro service's local datastore *in general* is its materialized view, which is used to serve its requests, not to persist event log. There are message brokers which provide you with the event log directly with customizable retention polices (sure, you can use database as event log aswell, again whats the point of putting load on your DB which needs to serve requests when broker already stored this information previously and allows you to set retention policies which can be replayed later at any time?) 2. It is not true that migrating a message flow -> Request-response paradigm is hard. Its infact quiet easy as you are making simple RPC calls. Sure, you will not have the same resiliency guarantees as in messaging as we are moving to request-response paradigm 3. Its not true that event log is mandatory. You might want to keep it if the business requires it. Else, easiest option is to use broker as a task distributor & consume the messages
This might help: www.techtarget.com/searchapparchitecture/tip/Event-driven-vs-message-driven-It-comes-down-to-complexity Basically, a message has a pre-defined receiver, while events are just published and multiple components can subscribe to them.
Hey Gaurav! First up... awesome video. So actually I work in the front office for an investment banking firm. So this architecture forms the whole core system of how we interact with things. If you would like to maybe we could have a conversation regarding the same it might help us both gain more perspective. Anyway. Cheerios! Keep posting
Hey Gaurav, Thanks a lot for this awesome video. The examples you give to explain something are awesome like counter strike, It so simplified the whole system. Thanks a lot :) Btw, is the event bus a message queue or any pubsub like Kafka?
"In Kafka message can be subscribed by multi consumers, means, many consumer types not many instances of same one. In a typical messaging queue like Rabbit a message can be consumed only once, and when consumed, the message disappears and isn’t accessible anymore." and as said in the video replicate and replay of messages is a feature of event driven architecture.
@@neshant89 Got the difference. Thanks :) Suppose in Kafka, for a single topic there are multiple subscribers ready to consume. When a message comes to the topic, all the consumers will receive the same message (copies) or any one of the consumers will only receive that message?
u are getting better each day :) loving your videos .... I would love to understand form you. How to do come up with a design solution in a system design? What are the factors you evaluate to add/optimize a system etc while designing a software system?
Thanks Pradip! Most of it comes from experience through work and projects. The interviewer focuses on certain points like consistency and availability, and we think of ways to achieve them. I'll try more detailed approaches in the videos 🙂
Thank you for all of your videos Gaurav. Great energy and excellent presentation skills. Keep it up. Although am newbie to distributed systems, I can see that the event-driven system is somewhat similar to saving snapshots in a file system or transaction journals in database which provides a point in time copy or state of it so that, user can later restore to any PIT if he needs. Correct me if my assumption is wrong.
@@gkcs Oh I see. Gaurav, one humble request. Can you please include Minutes of your presentation in the comments, summarizing topics covered with time in all of your videos, just as the events are timestamped in this system :) It will be of great help and time saver to jump to particular topic while reviewing or to refresh a particular topic, and, I am pretty sure everyone would want to watch these videos more than once.
What is the difference between pub-sub and event-driver architecture? From the above, I understood the Db log part for events (Which can be done in pub-sub too). Is pub-sub like a way to implement event-driver architecture?
great video, thanks for sharing. One thing I fee like might improve this is to show other similar or comparable systems. Some of benefits you are showing here is hard to be closely associated with event driven system, without comparing with other system, even briefly. I guess it is more due to the popularity of even driven system, I don't even know or say did not come up with other systems immediately when watching this video, then I thought isn't these benefits are from other system as well?
HI , Assume like we have one topic and two subscribers to that topic, How will the subscriber know that the message in the topic is already consumed or not , Will the message in the topic will be cleared after all the consumers consumed the message
Hi, nice video Gaurav! Just to be clear FPS doesn't work like that right? I know you just used it as an example.. but there must be some problem implementing EDD for FPS. If there is none..I don't know why I still have to suffer from latency issues in 2021. Am I missing something?
Followed from the previous video with the invoice example but this doesn’t give a solution to the distributed transaction across service boundary, I assume that’s just not possible to do here?
Hi Gaurav, Can you please explain how Git uses the event driven architecture or provide some material for that? One of my friends was asked this question in an interview recently.
@@gkcs Hi Gaurav, This is an awesome video but I think you can do it more comprehensive & Please make it more realistic with an example like u delivered the API video. Hope you will release it soon. All the Best thank you!
what if service A needs to know something that only service B knows or Process, Like repricing of a product in shopping cart. or availability check which can be an external gateway. Does this architecture poses lot of flaws in this particular case?
Hi Sir, If suppose I have an invoice app & I want to check stock, and invoice it, email the pdf. At last, the user wants to know the invoice is created or rejected due to low stock all the response should be sent to the front end for showing it after clicking invoice immediately. In this case, what is the best way to split as event-driven?
Wait. FPS games use UDP and P2P protocols for fast communications, right? I wonder does event-driven model on server side will have low enough latency for such checks.
What if the subscriber goes offline, and producer keeps sending events then the event bus will be overflowing and this might events loss. Looks like a disadvantage of event-based architecture.
@@gkcs we have timeout in req-resp model. If a service doesn't respond within timeout limits then we fail our request and handle it differently. But here, if a common event bus is used and one service(listener) is offline then the events of that service would fill the queue and block it for other services.
Hey Gaurav, Great Video. Good work :) Some questions that confused me.. 1. You mentioned about transactional guarantees in the advantages. You mentioned that it provides utmost-once and at-least once. How can a distributed system cannot guarantee utmost-once semantics unless your application handles it? 2. Some of the examples (Event log) that you gave especially described an event sourcing pattern where an application state is described as a series of events. Event sourcing is a specialized case, right? 3. "Difficult to know what people what to consume and what people don't wish to consume". Well, I did'nt get this at all. Can you give may be a simpler example of what you wish to convey by this? I think services can simply subscribe to topics they are interested in consuming and don't subscribe to the rest. Thanks and keep posting great content :)
Thanks Hjk! 1. At most once delivery is a natural consequence of distributed systems, yes. We use retries for at least once delivery. 2. Event driven means the events *drive* the behaviour of the system (events are sent as messages to other services). Event sourcing is more about storing all of the incremental changes in a system as a series of corresponding events. 3. Let's say you want to remove an API in your system, since you don't think other services are calling it. You find the number of times the API is hit, and make a decision (probably go to the callers and ask why they need the API). With events, it is hard to track who is consuming the event. The point of the system is to remove coupling between two services. But that also removes visibility of usage.
It is called chronical event sourcing It has many cons and one pros I hope no one consider to implement it into production instead take it as an option to takle mostly
Hi Gaurav, Thanks for the awesome videos....They are really helping to understand system designing. Can you please tell from where can I study more about System Designing. What all topics a good Software Engineer should know?
Really awesome video, thanks ..:), just a ques : Is Kafka also following event driven architecture, what's the difference btw pub sub and this architecture ?
Hi Gaurav, I'm trying to design Online-shoping with Event-driven Architecture but can't understand about message queue. How should I implement and why? E.g. If any customer orders a product, then a REST service would be called, which may or may not call other serivces like shipping, warehouse etc. then where comes message queue and how and where to implement this?
Great Videos! BTW you can maybe think of doing a series of reviews of popular pieces of software and software systems in terms of architecture and maybe classifying them on certain criterias (e.g. microservice vs monolith, request-response vs event-driven, etc.). I couldn't find a good source of such info on the Internet, so it is definitely a niche ;) Something similar to your "Design *insert popular app*" videos, but focused more on how and why it was done, comparison to other similar systems (for example Viber vs Whatsapp vs Messenger) and what the pros and cons are (for example in terms of scalability, performance, etc.)
I’m a bit confused about the difference between an event-driven vs a request-response architecture. Can’t you frame a “request“ as an “event”? So then you “publish” a request, pass it through to event bus, etc.
in request response you will have to wait for response... but in event-driven, you don't. you just assume, that once you have published an event, all the subscribers listening to it, will eventually finish up the task. but, There is no Guarantee in event-driven system. we can make the system fail-safe by adding up lots of fallback cases, but still, it doesn't always guarantee... and the fun part is, you will never know, if it failed ot succeeded... But, in the case of requst-response architecture, we will always know, if our request was success of failure. d
I am still new to learn all this. But from my experience, I've implemented some similar scenarios in my projects. So, my question is if I am a JavaScript Developer and my code is mostly written in Angular, then Reactive JS or RxJS is a event based library? Because I believe I can simply create Observables, Observer, Subject and Subscribe to them? I guess that's a event based system. Right?
Hey thanks again for another amazing video. Isn't reading/writing from DB time consuming? In gaming example where a millisecond is critical, I can see that DB access can be a bottle neck. Using CACHE memory from DB makes more sense.
Yes, caches are extensively used in low latency apps. The requests for a game will all probably map to a single server, so that the cache is consistent.
It's likely all the game state will just be stored on the game server. Just because you have this event system and microservices, doesn't mean you are communicating with them on the internet using ip, they could just be local sockets or files or direct RAM access or function calls etc.
Couple more points Your last few lines were “Node JS supports it”. Architecture has nothing to do with language/framework. One can use NodeJS, Python, Golang, java anything that supports EventBus tool (Kafka, Redis, etc) With this architecture, (micro)scalability can be a problem. Unless there’s a way to handle race-conditions between multiple subscribers to a channel, you may end up with duplicate data & processing. Or else, you are struck with single instances of a micro service for the rest of your life.
It depends on the requirements, but this is a useful component in many ticketing systems. Ticket creation, status updates, resolving and deletion can be modelled as events. JIRA internally uses Git, which is an event driven system 😁
@@gkcs or would it be better to use a messaging queue system? I guess the main requirements would be transaction guarantee, assign tasks to multiple types of work queues, be able to create new types of queues if needed, and be able to quickly search across all tickets.. would love to hear some suggestions :)
If you are preparing for a system design interview, try get.interviewready.io.
All the best 😁
If i subscribe, I will start listening to new events produced by your channel :)
One good thing about your videos is that you smile often, I feel that is much needed for the audience to be engaged through out your videos.
This channel is a gold mine.
Interviewer: what's the most important thing you want in your system design?
Me after watching this vid: You want headshots!
the title is soooo confusing!! The cover photo says "Event Sourcing" and the title says "Event Driven". I believe they are quite different things. Now you helped me mixed them up, thank you :)
So not exactly, his architecture diagram depicts a service that captures the events in an event store as it's dispatched on the bus which is event sourcing.
Event driven architecture version, is just the process of using events to communicate between services via event bus whether internally or externally and have each service subscribing to that event to later process it.
What he hasn't covered in depth was replaying events, but he briefly mentioned stream processing when he mentioned rollback when he gave examples and advantages.
Guys this is an overview video not an in-depth one. I believe he did a great job. Good man keep it up!
Its pretty cool of you to put this out! I might not pick up everything the first time, but its really elluminating and I really think about trying out new tech... thanks for that.
Thank you!
These videos are really good. More better, your excitement explaining things. There are lot of videos in youtube which may be better than your but man they are boring like hell that I get sleepy. Keep up the good work!
You have mixed Event-Driven and Event-Sourcing!
Event sourcing utilizes event driven architectures. He was simply giving a use case for it.
To explain event-driven architecture.... Would be helpful to not add in another layer on top in the example one is providing. That's what event sourcing does.
Wow your explanations are absolutely brilliant!
I believe why people like your video's is the way you enjoy the concepts and complete involve and explain them.. we can see in your eyes ( joy of happiness ) ... It could be great if you can show a video to community .. system design discussion with in group of people .. like everyone present their idea and how will you figure out the best in that room ...
A design discussion? That sounds like a good idea 🙂
I never miss your system design videos! I always wait for ur next video on system design.
Thanks!
Great explanation, thank you! 😌
Hi Gaurav,
First of all, thank you so much for the content which you have highlighted on the "Event-Driver Architecture".
One of the disadvantages which you have highlighted was "unable to track the end-2-end flow" in this architecture. So to answer that we can use the "Orchestration" pattern and you might be aware. So, basically, design the "workflow-orchestration" as soon as you get the service response so that we'll come to know where it hangs in case of any failure. We're using "CAMUNDA" for the same.
Thanks!
Good diagram - it is very well explain
Great video.. nicely explained.. you are best youtuber of this genre..
Thank you so much for the rich content.
Very interesting point you mentioned is that there is problem in understanding the flow of the code. I too faced myself this issue.
It's magic!
Great explanation - thank you!
You are welcome!
The way you described persisting events in the service makes it sound like an append file rather than database
Thanks Gaurav. NIce video. May I also point out a few advantages of event driven architectures?
1. It is the only way in which you can implement SEDA and hence scale individual stages as load increases. As the EDA works in eventual consistency model, the scale up times are not as relevant (or as relevant as they may be in req/resp paradigm)
2. This is necessary if you are implementing a CQRS based system.
3. Lambda architecture is only supported with events (allowing us to arguably beat the CAPs theoren)
4. In container world, where a container can be evicted any time (in the middle of a transaction etc), we either need all services to be idempotent or have an event driven architecture in place. Idempotency is not always possible.
Regards
I think this is more of event sourcing architecture (which is sub category of event driven) than event driven itself. In event driven architecture, my understanding is that there is no "replay" function nor there is implementation to support the replay functionality. A good example of it is almost all front end frameworks use event driven archs and they don't store any state
There is feature of replay subjects in Rxjs( Angular uses this) . It stores the previous States if I am not wrong 🤔
Observables is event-driven and it has replaying abilities.
I love your content! Thank you for sharing your videos!
great video thanks so much. you could talk at length, endlessly, about event driven architecture, and this video actually was the perfect level of depth.
event driven architecture, or subsets of it, are absolutely used in just about every single tech company these days. however, one of the hardest parts is figuring out what happens downstream after a message is published. can require major logging tools on top of entire systems like splunk or ELK to trace what happens through your systems, which is complex on it's own. i've spent days or weeks tracing transactions through microservice queues trying to solve bugs. Also, enforcing message schemas is another huge problem. rabbitMQ / kafka + nodeJS is a particularly nasty combination when it comes to debugging LOL. but i suppose thats why we get paid the big bucks ;)
Hey Gaurav! Your videos are great. Every topic has been explained to the perfect depth without making it boring. Just a minor concern I have, can you please order your videos correctly like sometimes a video that's coming up next is something that I don't know anything about which I know you have explained in some other video but don't know which one. It'll be great if you do that! Otherwise everything's great!
Thanks Eshika, noted!
You are too good with your explanation Gaurav.
Thank you!
i like your presentation skills!👌
Thank you!
@gaurav sen- enjoy watching your videos. You are very knowledgeable. Not many recent videos. you should do more. all the best
2023, still awesome,
Thank you!
Gaurav,
Multiple observations
1. You seemed to have confused event log with local db for each service. The micro service's local datastore *in general* is its materialized view, which is used to serve its requests, not to persist event log. There are message brokers which provide you with the event log directly with customizable retention polices (sure, you can use database as event log aswell, again whats the point of putting load on your DB which needs to serve requests when broker already stored this information previously and allows you to set retention policies which can be replayed later at any time?)
2. It is not true that migrating a message flow -> Request-response paradigm is hard. Its infact quiet easy as you are making simple RPC calls. Sure, you will not have the same resiliency guarantees as in messaging as we are moving to request-response paradigm
3. Its not true that event log is mandatory. You might want to keep it if the business requires it. Else, easiest option is to use broker as a task distributor & consume the messages
Excellent points, thanks for posting them 👍
@@gkcs That was quick :) . Again, Thanks for your effort in creating these videos. As an introductory videos, they are great
This is a great video. Thankyou so much gaurav.
Great conetent! It would be more helpful if you explain with examples of real-world applications.
Hi Gaurav, can you please explain what is the difference between message queues and event driven system.
This might help: www.techtarget.com/searchapparchitecture/tip/Event-driven-vs-message-driven-It-comes-down-to-complexity
Basically, a message has a pre-defined receiver, while events are just published and multiple components can subscribe to them.
Hey Gaurav! First up... awesome video. So actually I work in the front office for an investment banking firm. So this architecture forms the whole core system of how we interact with things. If you would like to maybe we could have a conversation regarding the same it might help us both gain more perspective. Anyway. Cheerios! Keep posting
Hey Gaurav, Thanks a lot for this awesome video. The examples you give to explain something are awesome like counter strike, It so simplified the whole system. Thanks a lot :)
Btw, is the event bus a message queue or any pubsub like Kafka?
Thanks!
A pubsub like kafka is how I look at it. Although a message queue will also do.
"In Kafka message can be subscribed by multi consumers, means, many consumer types not many instances of same one.
In a typical messaging queue like Rabbit a message can be consumed only once, and when consumed, the message disappears and isn’t accessible anymore."
and as said in the video replicate and replay of messages is a feature of event driven architecture.
@@neshant89 Got the difference. Thanks :)
Suppose in Kafka, for a single topic there are multiple subscribers ready to consume. When a message comes to the topic, all the consumers will receive the same message (copies) or any one of the consumers will only receive that message?
@@AmulyaSahoo349 All the consumers will receive the same set of messages
You have the best content
u are getting better each day :) loving your videos ....
I would love to understand form you. How to do come up with a design solution in a system design?
What are the factors you evaluate to add/optimize a system etc while designing a software system?
Thanks Pradip! Most of it comes from experience through work and projects. The interviewer focuses on certain points like consistency and availability, and we think of ways to achieve them. I'll try more detailed approaches in the videos 🙂
Need more videos with real time examples with micro service architecture
great job Gaurav
"I''ll cache you next time"
Cache me outside
How can I image React or Node.js using EDA? What are the events here? Do we have message queues/brokers as well here?
i like presentation skills!! 👌
Thank you for all of your videos Gaurav. Great energy and excellent presentation skills. Keep it up. Although am newbie to distributed systems, I can see that the event-driven system is somewhat similar to saving snapshots in a file system or transaction journals in database which provides a point in time copy or state of it so that, user can later restore to any PIT if he needs. Correct me if my assumption is wrong.
Thanks!
Somewhat, yes. The persistence is of commands/events instead of the snapshot though
@@gkcs Oh I see. Gaurav, one humble request. Can you please include Minutes of your presentation in the comments, summarizing topics covered with time in all of your videos, just as the events are timestamped in this system :) It will be of great help and time saver to jump to particular topic while reviewing or to refresh a particular topic, and, I am pretty sure everyone would want to watch these videos more than once.
Thanks, great work
What is the difference between pub-sub and event-driver architecture? From the above, I understood the Db log part for events (Which can be done in pub-sub too). Is pub-sub like a way to implement event-driver architecture?
Java vertex also uses evenbus architectures
Great content!
This is more of event sourcing. Event notifications can however work without event sources IMO
I would def like to see a video that explains the event driven architecture behind React, Git etc.
great video, thanks for sharing. One thing I fee like might improve this is to show other similar or comparable systems. Some of benefits you are showing here is hard to be closely associated with event driven system, without comparing with other system, even briefly. I guess it is more due to the popularity of even driven system, I don't even know or say did not come up with other systems immediately when watching this video, then I thought isn't these benefits are from other system as well?
Good point :)
What is event consumer and event consumer buffer size? There is a use of this in circuitbreaker design pattern but it is not clear what this does
Do we have any hands on code example for this architecture on youtube ??
Hi Gaurav , likewise for gaming scenario , how i can test and verify that a user has joined the request event created by me to join the playroom ?
i didn't know react uses this architecture. thanks.
I don't think react specifically uses this architecture? I know redux does and many react apps use redux
MaxPicAxe I think react use it on Reactdom features
Could someone please highlight the main difference between Pub-Sub and Event-Driven Architecture?
Wondering if Jenkins uses event-driven arch
HI , Assume like we have one topic and two subscribers to that topic, How will the subscriber know that the message in the topic is already consumed or not , Will the message in the topic will be cleared after all the consumers consumed the message
Hi, nice video Gaurav!
Just to be clear FPS doesn't work like that right? I know you just used it as an example.. but there must be some problem implementing EDD for FPS. If there is none..I don't know why I still have to suffer from latency issues in 2021. Am I missing something?
Why not just keep a server synchronised time counter in each client?
3:06 Wouldn't assigning a time stamp with each event solve this problem?
Followed from the previous video with the invoice example but this doesn’t give a solution to the distributed transaction across service boundary, I assume that’s just not possible to do here?
It is, but it needs consensus. Have a look at Raft.
What is an event bus? Is it a message queue?
Yes 😁
thanks for this video
Hi Gaurav, Can you please explain how Git uses the event driven architecture or provide some material for that? One of my friends was asked this question in an interview recently.
I'll get this out soon. Try the git internals book, although it's very large...
@@gkcs Hi Gaurav, This is an awesome video but I think you can do it more comprehensive & Please make it more realistic with an example like u delivered the API video. Hope you will release it soon. All the Best thank you!
what if service A needs to know something that only service B knows or Process, Like repricing of a product in shopping cart. or availability check which can be an external gateway. Does this architecture poses lot of flaws in this particular case?
Hi Sir, If suppose I have an invoice app & I want to check stock, and invoice it, email the pdf. At last, the user wants to know the invoice is created or rejected due to low stock all the response should be sent to the front end for showing it after clicking invoice immediately. In this case, what is the best way to split as event-driven?
Wait. FPS games use UDP and P2P protocols for fast communications, right? I wonder does event-driven model on server side will have low enough latency for such checks.
It's Kind of broadcasting which I come to know by any publisher side but tell me one thing .... Consumer will register itself somewhere or not?
What if the subscriber goes offline, and producer keeps sending events then the event bus will be overflowing and this might events loss. Looks like a disadvantage of event-based architecture.
And how would you avoid this with the request response model?
@@gkcs we have timeout in req-resp model. If a service doesn't respond within timeout limits then we fail our request and handle it differently. But here, if a common event bus is used and one service(listener) is offline then the events of that service would fill the queue and block it for other services.
@@shubhamagrawal8478 Distribute and scale the queue.
Hey Gaurav,
Great Video. Good work :)
Some questions that confused me..
1. You mentioned about transactional guarantees in the advantages. You mentioned that it provides utmost-once and at-least once. How can a distributed system cannot guarantee utmost-once semantics unless your application handles it?
2. Some of the examples (Event log) that you gave especially described an event sourcing pattern where an application state is described as a series of events. Event sourcing is a specialized case, right?
3. "Difficult to know what people what to consume and what people don't wish to consume". Well, I did'nt get this at all. Can you give may be a simpler example of what you wish to convey by this? I think services can simply subscribe to topics they are interested in consuming and don't subscribe to the rest.
Thanks and keep posting great content :)
Thanks Hjk!
1. At most once delivery is a natural consequence of distributed systems, yes. We use retries for at least once delivery.
2. Event driven means the events *drive* the behaviour of the system (events are sent as messages to other services). Event sourcing is more about storing all of the incremental changes in a system as a series of corresponding events.
3. Let's say you want to remove an API in your system, since you don't think other services are calling it. You find the number of times the API is hit, and make a decision (probably go to the callers and ask why they need the API).
With events, it is hard to track who is consuming the event. The point of the system is to remove coupling between two services. But that also removes visibility of usage.
@@gkcs Quick reply, Thank you. Makes sense :)
It is called chronical event sourcing
It has many cons and one pros
I hope no one consider to implement it into production instead take it as an option to takle mostly
Hi Gaurav, Thanks for the awesome videos....They are really helping to understand system designing. Can you please tell from where can I study more about System Designing. What all topics a good Software Engineer should know?
Hey Ashish, thanks! You could try the highscalability blog and tech conference videos on TH-cam 😁
Really awesome video, thanks ..:), just a ques : Is Kafka also following event driven architecture, what's the difference btw pub sub and this architecture ?
I made a video on publisher subscriber architectures here:
th-cam.com/video/FMhbR_kQeHw/w-d-xo.html
Hi Gaurav, I'm trying to design Online-shoping with Event-driven Architecture but can't understand about message queue. How should I implement and why? E.g. If any customer orders a product, then a REST service would be called, which may or may not call other serivces like shipping, warehouse etc. then where comes message queue and how and where to implement this?
Use aws sqs service or sns service
Is it better to maintain a event router service or every service runs its own instance of the subscriber service. ?
Have a look at what a service mesh is. I will be speaking about this in detail sometime soon :)
Kind of State design pattern partly.. You seem very happy though
WE WANT HEADSHOTS!!!!!
does event poll accepts multiple requests at a same time. I mean will it process multiple requests in concurrent?
Is there any role of msg broker in this architecture
Event logs replaying is actually what event sourcing architecture is.
Great Videos!
BTW you can maybe think of doing a series of reviews of popular pieces of software and software systems in terms of architecture and maybe classifying them on certain criterias (e.g. microservice vs monolith, request-response vs event-driven, etc.). I couldn't find a good source of such info on the Internet, so it is definitely a niche ;) Something similar to your "Design *insert popular app*" videos, but focused more on how and why it was done, comparison to other similar systems (for example Viber vs Whatsapp vs Messenger) and what the pros and cons are (for example in terms of scalability, performance, etc.)
amazing
Re. 1:19 another application to add your list is Algorithm trading
Hi gaurav , i like the architech and i appreciate that, could you please share a architecture model of e-commerce order management micro service
Can you make a video explaining event loop in node
Kafka is great building event driven system
Nice.. I need a example for database table structure in event driven architecture.
Event bus may be MQ, right???
Yes.
how React and node js use event driven architecture?
I’m a bit confused about the difference between an event-driven vs a request-response architecture. Can’t you frame a “request“ as an “event”? So then you “publish” a request, pass it through to event bus, etc.
in request response you will have to wait for response... but in event-driven, you don't. you just assume, that once you have published an event, all the subscribers listening to it, will eventually finish up the task.
but, There is no Guarantee in event-driven system. we can make the system fail-safe by adding up lots of fallback cases, but still, it doesn't always guarantee... and the fun part is, you will never know, if it failed ot succeeded...
But, in the case of requst-response architecture, we will always know, if our request was success of failure.
d
@@neerpatel4647 ahh cool makes sense, thank you!
I see that you described event sourcing here. Does event driven architecture always comes with event sourcing?
From what I understand, yes.
I am still new to learn all this. But from my experience, I've implemented some similar scenarios in my projects. So, my question is if I am a JavaScript Developer and my code is mostly written in Angular, then Reactive JS or RxJS is a event based library? Because I believe I can simply create Observables, Observer, Subject and Subscribe to them? I guess that's a event based system. Right?
Hey Prateek! That is true... at least at the code level.
Event driven systems produce and consume events across microservices too.
Wow! I was thinking of implementing a notification service. Maybe after watching this video I'm closer to what i want to achieve.
Great explain but i need listen video at 0.75 speed 😬
Lol I watch at 1.75 speed :)
Hey thanks again for another amazing video. Isn't reading/writing from DB time consuming? In gaming example where a millisecond is critical, I can see that DB access can be a bottle neck. Using CACHE memory from DB makes more sense.
Yes, caches are extensively used in low latency apps. The requests for a game will all probably map to a single server, so that the cache is consistent.
This is why Redis is so popular. It's an: "in-memory data structure store, used as a database, cache and message broker."
It's likely all the game state will just be stored on the game server. Just because you have this event system and microservices, doesn't mean you are communicating with them on the internet using ip, they could just be local sockets or files or direct RAM access or function calls etc.
Bro where do you work?
Uber.
@@gkcs thanks. You are great!
Why do we need to replay events while replacing service we could have just copied all the data
Couple more points
Your last few lines were “Node JS supports it”. Architecture has nothing to do with language/framework. One can use NodeJS, Python, Golang, java anything that supports EventBus tool (Kafka, Redis, etc)
With this architecture, (micro)scalability can be a problem. Unless there’s a way to handle race-conditions between multiple subscribers to a channel, you may end up with duplicate data & processing. Or else, you are struck with single instances of a micro service for the rest of your life.
Can you recommend a book for system design for beginners?
Nope. Don't know any.
The closest you can get is 'Designing data intensive applications', but that's not a book for beginners.
With great availability, comes bad consistency
This is more of an Event Sourcing
just for Counter strike example. I understood it :)
would this be a good architecture for a Customer Ticket System?
It depends on the requirements, but this is a useful component in many ticketing systems.
Ticket creation, status updates, resolving and deletion can be modelled as events. JIRA internally uses Git, which is an event driven system 😁
@@gkcs or would it be better to use a messaging queue system? I guess the main requirements would be transaction guarantee, assign tasks to multiple types of work queues, be able to create new types of queues if needed, and be able to quickly search across all tickets.. would love to hear some suggestions :)
@@luisarellano1522 This is an offshoot of the message queuing system.
The events are effectively messages.