Interesting extension of the outbox pattern in case of higher volumes or multiple subscribers. Also give the possibility to add a new subscriber and give them access to desired history of events. So no hate of outbox, but extra options in case it is needed. Always good to have options. 😊
Another issue, which for me is more important, is "event format evolution" and the associate timescale at which the event should be valid and useful. All to often, the program internal events are simply serialized and pushed out, without considering versioning and other future-proofing (identities, composition or not, collections or not, aggregation or not, ++). And since it is so easy to just serialize, push and forget not much love is given until years later when some backward-compatibility issue arises because the "forward-compatibility" concern was ignored.
This implementation looks easy to implement but there are some challanges, hardest one is Gap problem. If you use auto incremented id or sequence in id column, simultaneous transactions can create ordering problem so you can see onordered ids when querying from this table
We have to be careful to not mix and match too many requirements, and create databases that try to serve them all. In many circumstances you would hit limitations of the type of database used for the domain transactions, which requires to move the event consumption to more suitable technologies. In such scenario the common outbox pattern is usually sufficient as you would publish the event (once more) somewhere else, bringing all the conveniences you describe in a solid, scalable and sustainable way 🤔 I see that you didn't say to use the same database nor technology to achieve what you describe, but I think many people will interpret as such. For me the outbox pattern is to explain people the problems of multiple transaction, and how it can be solved. Consumption and processing of events is another beast.
You seem to have missed the whole point, which is 💯 about storing events long-term for reuse. If you don't do that, you aren't thinking far enough in the future.
First of all, any sensible outbox implementation deals with table offsets and never-ever delete anything or using any kind of statuses, at least with the tx context. As soon as you have offsets, storing them in db table is a no brainer. But even though, it remaining the same outbox, IMO.
"A separate service is in charge of regularly scanning the outbox table for new events, sending them to Amazon SQS, and deleting them from the table if Amazon SQS responds successfully." docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html#transactional-outbox-intent
Hi Vernon, I think, your solution is not an alternative, just an improvement. Main goal in transactional outbox pattern is publish messages after commit. Yes, new solutions, comes with new challenges. I use this pattern with transaction log tailing or if it is not an option then I use polling publisher method in order to collect events from table or collection. After your video, I do not hate outbox pattern fortunately 😅. Thanks for an insightful video.
In some human languages, hate can mean "to love less." Still, the Outbox deletes or marks elements (rows) sent, which makes them ephemeral. I'm amazed how few people understand the power of long-lived domain events, and that send-once is hardly the point of using domain events in the first place. So many comments indicate a failure to understand that essential requirement.
BTW, my suggestion is an alternative, and is absolutely not the Outbox pattern. But don't take my word for it. "A separate service is in charge of regularly scanning the outbox table for new events, sending them to Amazon SQS, and deleting them from the table if Amazon SQS responds successfully." docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html#transactional-outbox-intent-intent
Interesting extension of the outbox pattern in case of higher volumes or multiple subscribers.
Also give the possibility to add a new subscriber and give them access to desired history of events.
So no hate of outbox, but extra options in case it is needed. Always good to have options. 😊
Another issue, which for me is more important, is "event format evolution" and the associate timescale at which the event should be valid and useful. All to often, the program internal events are simply serialized and pushed out, without considering versioning and other future-proofing (identities, composition or not, collections or not, aggregation or not, ++). And since it is so easy to just serialize, push and forget not much love is given until years later when some backward-compatibility issue arises because the "forward-compatibility" concern was ignored.
That's an important but different topic. I could cover it in the future.
This implementation looks easy to implement but there are some challanges, hardest one is Gap problem. If you use auto incremented id or sequence in id column, simultaneous transactions can create ordering problem so you can see onordered ids when querying from this table
Solved.
github.com/vlingo/xoom-symbio/tree/master/src/main/java/io/vlingo/xoom/symbio/store/gap
We have to be careful to not mix and match too many requirements, and create databases that try to serve them all. In many circumstances you would hit limitations of the type of database used for the domain transactions, which requires to move the event consumption to more suitable technologies. In such scenario the common outbox pattern is usually sufficient as you would publish the event (once more) somewhere else, bringing all the conveniences you describe in a solid, scalable and sustainable way 🤔
I see that you didn't say to use the same database nor technology to achieve what you describe, but I think many people will interpret as such. For me the outbox pattern is to explain people the problems of multiple transaction, and how it can be solved. Consumption and processing of events is another beast.
You seem to have missed the whole point, which is 💯 about storing events long-term for reuse. If you don't do that, you aren't thinking far enough in the future.
I really like your content. However, something is wrong with the sound. I can only hear it on my left ear.
I'll have to see if it was processed for mono, but I was fairly sure it was stereo. Thanks for watching.
What about letting the subscriber keep track of the last event consumed?
Could do. The point is about journaling rather than single-send and delete (hard or soft).
First of all, any sensible outbox implementation deals with table offsets and never-ever delete anything or using any kind of statuses, at least with the tx context. As soon as you have offsets, storing them in db table is a no brainer. But even though, it remaining the same outbox, IMO.
Where did you get your definition? What you describe is not the Outbox pattern.
"A separate service is in charge of regularly scanning the outbox table for new events, sending them to Amazon SQS, and deleting them from the table if Amazon SQS responds successfully."
docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html#transactional-outbox-intent
Hi Vernon, I think, your solution is not an alternative, just an improvement. Main goal in transactional outbox pattern is publish messages after commit. Yes, new solutions, comes with new challenges.
I use this pattern with transaction log tailing or if it is not an option then I use polling publisher method in order to collect events from table or collection.
After your video, I do not hate outbox pattern fortunately 😅.
Thanks for an insightful video.
In some human languages, hate can mean "to love less." Still, the Outbox deletes or marks elements (rows) sent, which makes them ephemeral. I'm amazed how few people understand the power of long-lived domain events, and that send-once is hardly the point of using domain events in the first place. So many comments indicate a failure to understand that essential requirement.
BTW, my suggestion is an alternative, and is absolutely not the Outbox pattern. But don't take my word for it.
"A separate service is in charge of regularly scanning the outbox table for new events, sending them to Amazon SQS, and deleting them from the table if Amazon SQS responds successfully."
docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html#transactional-outbox-intent-intent