Adam here. Event-driven architecture (EDA) can mean different things to different people. They can be internal to a single application, a distributed network of asynchronous applications, or a queue of tasks for a worker to process. But these all fall under the event-driven umbrella. With this video, I wanted to highlight four main types of EDA to illustrate how each type plays a role in building resilient, scalable, and performant architectures. Let me know what you think of this video in the comments below. Also, would love to hear what topics on event-driven architecture you would like to see covered next!
Fantastic, thank you! Currently reading your book too. And I mean I am recording myself narrating your book, lol. Of course, not for anyone but me, but it really helps my brain conceptualize and retain the information better.
Congratulations on the content Adam, this area of technology of ours is very complicated, with many of the same terms having different meanings depending on the context, I'm studying postgraduate studies in software architecture and it drives me crazy. Taking advantage, when we talk about an ephemeral form without retention, we are referring to a pattern where events are processed in real time and are not stored for later consultation, is that it?
Ephemeral means that there is no indefinite persistence of the events. It can vary from "all events are only in memory" (and therefore vulnerable to deletion upon power loss) to "events are persisted to disk, but are deleted after a short period of time "(seconds, minutes, or perhaps after consumption). For example, a consumer may only receive the messages if it is online at the time the producer writes it to the broker. A consumer coming online later is unable to access the historic events because they are no longer stored anywhere.
Nothing stops you from doing it, however it all depends on the use case. For example, if your app (assuming you’re referring to a ‘service’) is a monolith running as a single instance (no horizontal scaling anticipated or supported), then in-memory queue would be a first option to consider - in general it’s simpler and cheaper. However if a service runs in multiple instances and you already have Kafka as part of your infrastructure - it’s perfectly fine to set up inter-instance communication via Kafka. Here’s a couple of examples I’ve seen in real production: - Task breakdown: receive an external event, split it into N atomic independent pieces and sent to all service instances to parallelize work (shared consumer group) - State propagation (local cache): instance receiving a write forwards it into an internal topic, all instances of the service use unique consumer groups to receive copy of this message (fan-out) - analogue of sticky sessions: relying on partitioning algorithm (ex: userId) to make sure all events belonging to the same logical entity (here - user) are getting processed by the same service instance (be careful with rebalancing and choosing right partition key)
Wade from Confluent here. You definitely can use Kafka for internal communication. That's the foundation of a pattern known as the "Listen to Yourself" pattern (more information here: th-cam.com/video/If2W6tmDn80/w-d-xo.html). Furthermore, when using Event Sourcing and CQRS, it's not unreasonable to have the read models for your service built using events that come from the write model after they are published to Kafka. So, yes, there are plenty of use cases where you can do this. In this video, Adam is isolating the technologies that best represent the specific patterns. So he's not saying you can't use the technologies in other scenarios. You definitely can. Rather, he's trying to say that Kafka is a typical example of Pub/Sub, ActiveMQ is a typical example of Queuing, Akka is a typical example of internal, etc. Each of these technologies can do more, but their most common usage fits into one category or another.
Wade here, from Confluent. We actually had to scour the world for someone who could write backward and forward so that we could make this video. It's a very rare talent practiced only by a small community in Canada...Ok, actually, he just writes normally on something called a "Light Board" and then they flip the image in post-production.
Good explanation - not quite correct examples. ESB is not about ephemeral messaging (if you worked with it you would know not from theory - I did for years). Ephemeral is topic related. However, Apache Camel ESB with Apache MQ GUARANTEES delivery even on topic provided that subscriber subscribed once (it is lost in two cases: not subscribed first time or you exceed parameters of how for long it is stored).
Adam here. I admit I waffled on putting "ESB" into that category. You're quite right that an Apache Camel ESB / MQ guarantees delivery aside from those two cases you mention. The thing that makes ESBs difficult to talk about is that there isn't a global standard, so what you described (to me) is an ESB with an enhanced persistence layer. Using my 4 types of EDA framework, I would say it's ephemeral in the sense that the record is lost if you aren't subscribed when it happens, or it times out (as you put it). But it's also queue-based as you're using durable queues to store the messages. I guess that's the hard part about trying to make 4 neat categories - real life systems and use cases rarely fall nicely into just one :)
@@ConfluentDevXTeam Hi Adam. I liked your explanation very much. I was reading and listening to Confluent and watching and learning Kafka for last years (wondering if you should somehow join Confluent perhaps ;) I even found one problem on state race condition in microservices if someone is not careful enough in designing and implementation with Kafka events (related to Saga pattern). But Kafka for many is the way to go and I was on architecture while in Fiserv that uses Kafka in AWS for credit cards (very nice design, but not mine). So Kafka and Confluent works are very close to my heart, but I hate to use my prior intimate knowledge with other systems to point some gaps. Thanks for correction.
EDA is by nature the most powerful approach for distributed systems. However, this video is about structure and a bit of behavior pattern. If you need more info, you will need to research QoS, versioning, schema ...
Adam here. Event-driven architecture (EDA) can mean different things to different people. They can be internal to a single application, a distributed network of asynchronous applications, or a queue of tasks for a worker to process. But these all fall under the event-driven umbrella.
With this video, I wanted to highlight four main types of EDA to illustrate how each type plays a role in building resilient, scalable, and performant architectures.
Let me know what you think of this video in the comments below. Also, would love to hear what topics on event-driven architecture you would like to see covered next!
Fantastic, thank you! Currently reading your book too.
And I mean I am recording myself narrating your book, lol. Of course, not for anyone but me, but it really helps my brain conceptualize and retain the information better.
Good stuff Adam!
Congratulations on the content Adam, this area of technology of ours is very complicated, with many of the same terms having different meanings depending on the context, I'm studying postgraduate studies in software architecture and it drives me crazy. Taking advantage, when we talk about an ephemeral form without retention, we are referring to a pattern where events are processed in real time and are not stored for later consultation, is that it?
Ephemeral means that there is no indefinite persistence of the events. It can vary from "all events are only in memory" (and therefore vulnerable to deletion upon power loss) to "events are persisted to disk, but are deleted after a short period of time "(seconds, minutes, or perhaps after consumption). For example, a consumer may only receive the messages if it is online at the time the producer writes it to the broker. A consumer coming online later is unable to access the historic events because they are no longer stored anywhere.
Super helpful!!
cant you use kafka for app internal communication or queues aswell? or is this not recommended?
Nothing stops you from doing it, however it all depends on the use case. For example, if your app (assuming you’re referring to a ‘service’) is a monolith running as a single instance (no horizontal scaling anticipated or supported), then in-memory queue would be a first option to consider - in general it’s simpler and cheaper.
However if a service runs in multiple instances and you already have Kafka as part of your infrastructure - it’s perfectly fine to set up inter-instance communication via Kafka. Here’s a couple of examples I’ve seen in real production:
- Task breakdown: receive an external event, split it into N atomic independent pieces and sent to all service instances to parallelize work (shared consumer group)
- State propagation (local cache): instance receiving a write forwards it into an internal topic, all instances of the service use unique consumer groups to receive copy of this message (fan-out)
- analogue of sticky sessions: relying on partitioning algorithm (ex: userId) to make sure all events belonging to the same logical entity (here - user) are getting processed by the same service instance (be careful with rebalancing and choosing right partition key)
Wade from Confluent here. You definitely can use Kafka for internal communication. That's the foundation of a pattern known as the "Listen to Yourself" pattern (more information here: th-cam.com/video/If2W6tmDn80/w-d-xo.html). Furthermore, when using Event Sourcing and CQRS, it's not unreasonable to have the read models for your service built using events that come from the write model after they are published to Kafka. So, yes, there are plenty of use cases where you can do this.
In this video, Adam is isolating the technologies that best represent the specific patterns. So he's not saying you can't use the technologies in other scenarios. You definitely can. Rather, he's trying to say that Kafka is a typical example of Pub/Sub, ActiveMQ is a typical example of Queuing, Akka is a typical example of internal, etc. Each of these technologies can do more, but their most common usage fits into one category or another.
Bro how do you write mirrored like that on glass?
Wade here, from Confluent. We actually had to scour the world for someone who could write backward and forward so that we could make this video. It's a very rare talent practiced only by a small community in Canada...Ok, actually, he just writes normally on something called a "Light Board" and then they flip the image in post-production.
Good explanation - not quite correct examples. ESB is not about ephemeral messaging (if you worked with it you would know not from theory - I did for years). Ephemeral is topic related. However, Apache Camel ESB with Apache MQ GUARANTEES delivery even on topic provided that subscriber subscribed once (it is lost in two cases: not subscribed first time or you exceed parameters of how for long it is stored).
Adam here. I admit I waffled on putting "ESB" into that category. You're quite right that an Apache Camel ESB / MQ guarantees delivery aside from those two cases you mention. The thing that makes ESBs difficult to talk about is that there isn't a global standard, so what you described (to me) is an ESB with an enhanced persistence layer. Using my 4 types of EDA framework, I would say it's ephemeral in the sense that the record is lost if you aren't subscribed when it happens, or it times out (as you put it). But it's also queue-based as you're using durable queues to store the messages.
I guess that's the hard part about trying to make 4 neat categories - real life systems and use cases rarely fall nicely into just one :)
@@ConfluentDevXTeam Hi Adam. I liked your explanation very much. I was reading and listening to Confluent and watching and learning Kafka for last years (wondering if you should somehow join Confluent perhaps ;) I even found one problem on state race condition in microservices if someone is not careful enough in designing and implementation with Kafka events (related to Saga pattern). But Kafka for many is the way to go and I was on architecture while in Fiserv that uses Kafka in AWS for credit cards (very nice design, but not mine). So Kafka and Confluent works are very close to my heart, but I hate to use my prior intimate knowledge with other systems to point some gaps. Thanks for correction.
@@maciekskontakt Thank you for your insights too! I appreciate your kind words as well.
Hello from Brazil
EDA is by nature the most powerful approach for distributed systems. However, this video is about structure and a bit of behavior pattern. If you need more info, you will need to research QoS, versioning, schema ...