I generally consider ordered processing for most of the systems, be it a payment system or a chat box or an order management system or a weather notification system etc Limited usecases like "processing orders with abundant supply" accepts change in order By defining boundaries in an optimistic way, different tasks can be sent to different channels/topics and even with single consumer per channel/topic will be helpful
Thanks for the video I liked and subscribed but feel like you rushed the most important and complex topic 'processing out of order', the illustration was quite poor and you lost me there. please spend more time or give two examples when going into a complex topic. I have no idea what a saga is and have never heard of it, would be nice to define it. how to deal with failure gave me an idea, since kafka keeps a log would it be possible to code this: stop > consumer deletes messages received after failure > replay from the failed message using the log > continue as normal.
I have a bunch of different videos on workflows and different ways to execute them. Check out this video about sagas: th-cam.com/video/rO9BXsl4AMQ/w-d-xo.html
Thanks. There are some of message brokers which support message ordering feature: * Message sessions in Azure Service Bus, * Message group ID in Amazon SQS FIFO. What else do you guys know?
Very nice video as always Derek ! It seems to me that the saga is loaded the same way an aggregate is loaded in an event sourced system. Should a saga also have a key like the aggregate id ?
State is persisted usually in durable storage and is loaded when the saga is constructed. You basically map each event it handles to the ID that connects them all. In my example it was an OrderId.
So assuring message order shall be message broker matter,not SAGA worker because in this case we shall be embed some business logic in the saga worker, right? Thank you!
Thanks for the video, it brings an interesting perspective on how to avoid unnecessary additional technical complexity with in order processing! But there's still one thing I wonder about. It's evident that something is wrong with ordering if you miss a message that's required for your business flow. But what about optional messages? For example, if after an order is placed I change the delivery destination (DeliveryDestionationChanged event) and the flow doesn't process it, I might end up delivering to a wrong place. And considering that changing destination is an optional step, there's no easy way to notice that this step is skipped. Could you please share your experience on how to handle these situations?
That's a good example to illustrate that thinking about the wrong flow. Changing the delivery destination after an order is placed probably shouldn't start at the order (in sales) it should probably start with changing the destination in shipping/receiving and then flow back to sales.
hi Derek, what if the update events are sent to the consumer in the future just like the reserve-expire (delay-delivery) pattern you used in a previous video
Nice video as always, from what i have seen so far it's almost always avoidable problem, however, these are super ways to handle it when there's no way out :)
Thanks. Now I kindof understand the partition when creating topic in kafka.
Exactly.
I generally consider ordered processing for most of the systems, be it a payment system or a chat box or an order management system or a weather notification system etc
Limited usecases like "processing orders with abundant supply" accepts change in order
By defining boundaries in an optimistic way, different tasks can be sent to different channels/topics and even with single consumer per channel/topic will be helpful
Thank you! Appreciate all the efforts you put into these videos!
Glad you like them!
Thanks for the video I liked and subscribed but feel like you rushed the most important and complex topic 'processing out of order', the illustration was quite poor and you lost me there. please spend more time or give two examples when going into a complex topic. I have no idea what a saga is and have never heard of it, would be nice to define it.
how to deal with failure gave me an idea, since kafka keeps a log would it be possible to code this: stop > consumer deletes messages received after failure > replay from the failed message using the log > continue as normal.
I have a bunch of different videos on workflows and different ways to execute them. Check out this video about sagas: th-cam.com/video/rO9BXsl4AMQ/w-d-xo.html
Thanks.
There are some of message brokers which support message ordering feature:
* Message sessions in Azure Service Bus,
* Message group ID in Amazon SQS FIFO.
What else do you guys know?
Solace (full disclosure, they sponsor some of my videos, including this one 😀).
Very nice video as always Derek !
It seems to me that the saga is loaded the same way an aggregate is loaded in an event sourced system. Should a saga also have a key like the aggregate id ?
State is persisted usually in durable storage and is loaded when the saga is constructed. You basically map each event it handles to the ID that connects them all. In my example it was an OrderId.
So assuring message order shall be message broker matter,not SAGA worker because in this case we shall be embed some business logic in the saga worker, right? Thank you!
Thanks for the video, it brings an interesting perspective on how to avoid unnecessary additional technical complexity with in order processing!
But there's still one thing I wonder about. It's evident that something is wrong with ordering if you miss a message that's required for your business flow. But what about optional messages?
For example, if after an order is placed I change the delivery destination (DeliveryDestionationChanged event) and the flow doesn't process it, I might end up delivering to a wrong place. And considering that changing destination is an optional step, there's no easy way to notice that this step is skipped. Could you please share your experience on how to handle these situations?
That's a good example to illustrate that thinking about the wrong flow. Changing the delivery destination after an order is placed probably shouldn't start at the order (in sales) it should probably start with changing the destination in shipping/receiving and then flow back to sales.
Since network calls are involved during publishing how can we still maintain the ordering of events?
hi Derek, what if the update events are sent to the consumer in the future just like the reserve-expire (delay-delivery) pattern you used in a previous video
Generally want a version in the event or some way to indicate know if the event being processed is stale to what you already processed.
Thanks, very helpful !!!
Glad it helped!
Nice video as always, from what i have seen so far it's almost always avoidable problem, however, these are super ways to handle it when there's no way out :)
Great as always, Thanks Derek !
As far as I know rabbitmq does not natively support partition-key, do you have any trick?
Consistent hash exchange
Check out consistent hash exchange with single active consumer
Thanks man, very helpful!
Glad to hear it!
Loved it!
Thanks!