Wow, this is such an amazing speech. Thanks a lot for sharing your wisdom 👍👍 23:36 here is the summary of the lecture but it would not be so easy if you haven’t followed the whole journey
Why didn’t they ask any event failure type rollback questions in microservices !! This is a really great presentation !!Too much content here in approx 30 minutes !! I have gone through this 5 times !!! Excellent stufff !!
there isn't a fully exact way you can achieve complete data consistency with micro-service without a level of decoupling trust me. using some kind of locking mechanism and consumer acknowledgement could help to some extent though. for example, use ACID features to lock a transaction on your service db, send and event to a queue like rabbitmq or kafka and wait for acknowledgement before persisting your data in your service before unlocking or if error occurs then unlock and report the error. this same scenario for an event listener as well
@18:21, how will the "orders" event cancelled if payments did not go thru or vice-versa. In other words what happens if number 4 errors out in one of the 2 places
It would be a payment service responsibility to publish an event into Orders steam/topic to inform other services that event is canceled OR it would be an order service responsibility to monitor some Payments steam/topic to detect failed payment and cancel the order. Depending on where do you want to have this complexity and if you want payments service know a bit about orders or orders to know about the payment. Looks like payments already know about orders so 1st scenario looks slightly better to me
Or the Order event processor only responds to Payment Succeeded events in the first place. Or possibly it responds to the same Order Initiated event that triggers the Payment processor by creating/storing an Order in a pending / incomplete state. When it encounters a Payment Succeeded event, the Order event processor will load the pending order and advance it to an actionable state and possibly publish a new event that will signal a different processor to act upon. If the Order event processor receives a Payment Failed event, it will load the pending order and update it to a rejected state. This does introduce a potential race condition, however, that an Order event processor could encounter a Payment Succeeded event before one of its sibling processors has successfully finished processing the Order Initiated event, and hence there is no pending order for it to load and update. So I think the first approach may be the better one. But the downside of only responding to Payment Succeeded events is that if the payment failed, there will be no record of the Order existing at all, other than in the event log itself.
Hi Grygoriy Gonchar, thanks for the nice video on data consistency. I have a question : Where is the "Application Event Log" containing the order details (Request, Payment, Flight etc) stored ? I mean in which service ? In the example you mentioned, in "makemytrip" there are 5 services, where is the "Application Event Log" Maintained ? Is it maintained by a separate microservice (SAGA) or in one of the 5 microservices (Order, Payment, Hotel, Flight, Transfer) ?
Awesome presentation. No questions from the audience, made me surprise! Anyway, He mention in Kafka vs RabbitMQ, with Kafka there are no retries out of the box. How come that happen?
In Kafka, there is no easy way to retry processing a specific message without stop moving the cursor and blocking the whole partition processing. Sure you can retry processing a message easily by not moving the cursor forward, but that would delay processing the next messages in that partition. In contrast in message brokers like RabbitMQ you can do that.
What he said ^ I'll just add that different use cases have different requirements for strict ordering of event processing. In some applications, it is critical that the next message not be processed until after the current message has been processed, and hence the cursor should not be allowed to move and the whole partition's queue blocked until the issue is resolved. In other use cases, one or two retries can be attempted and then the failure recorded and the processing allowed to move on past the failed message. And in some use cases, a given event should only be processed one time and the queue continues, regardless of whether it succeeded or failed. All of these approaches are easily achieved using Kafka's offset counter / cursor. But the developer needs to make a conscious decision about which approach is appropriate for a their use case.
I agree ... and some of the patterns were not clear segregated. For example, he talked about reconciliation first ... but later seemed to talk about CQRS etc (stuff in the SAGA pattern). Is reconciliation part of the saga pattern or is it a different pattern altogether? Was confused on this.
Approaches for dealing with data inconsistency
1. saga
2. reconcilation
3. application event log
4. 2PC/XA
5. event first
Moving functions that require data consistency in the same Microservice was great information.
Data consistency is number one priority ;)
Haha I see what you did there ;)
Wow, this is such an amazing speech. Thanks a lot for sharing your wisdom 👍👍
23:36 here is the summary of the lecture but it would not be so easy if you haven’t followed the whole journey
Why didn’t they ask any event failure type rollback questions in microservices !! This is a really great presentation !!Too much content here in approx 30 minutes !! I have gone through this 5 times !!! Excellent stufff !!
Thanks for your warm feedback! And sorry for making it that concise, that's not just you who shared that.
In germany people tend to be quite. stupid developers who don't listen and yet make a mistake and leave the company
You can accomplish rollback with compensation
Thanks, Grygoriy, that's a great speech
there isn't a fully exact way you can achieve complete data consistency with micro-service without a level of decoupling trust me. using some kind of locking mechanism and consumer acknowledgement could help to some extent though. for example, use ACID features to lock a transaction on your service db, send and event to a queue like rabbitmq or kafka and wait for acknowledgement before persisting your data in your service before unlocking or if error occurs then unlock and report the error. this same scenario for an event listener as well
Эх, наших всегда можно распознать по акценту :)
Классное видео, большое спасибо :)
говорит на русском, английскими словами)
НАУ може пишатися такими випускниками як він
@18:21, how will the "orders" event cancelled if payments did not go thru or vice-versa. In other words what happens if number 4 errors out in one of the 2 places
It would be a payment service responsibility to publish an event into Orders steam/topic to inform other services that event is canceled OR it would be an order service responsibility to monitor some Payments steam/topic to detect failed payment and cancel the order. Depending on where do you want to have this complexity and if you want payments service know a bit about orders or orders to know about the payment. Looks like payments already know about orders so 1st scenario looks slightly better to me
@@grygoriygonchar6994 Isn't that similar to a compensating transaction in the saga pattern except that this is event driven ?
Or the Order event processor only responds to Payment Succeeded events in the first place.
Or possibly it responds to the same Order Initiated event that triggers the Payment processor by creating/storing an Order in a pending / incomplete state. When it encounters a Payment Succeeded event, the Order event processor will load the pending order and advance it to an actionable state and possibly publish a new event that will signal a different processor to act upon. If the Order event processor receives a Payment Failed event, it will load the pending order and update it to a rejected state.
This does introduce a potential race condition, however, that an Order event processor could encounter a Payment Succeeded event before one of its sibling processors has successfully finished processing the Order Initiated event, and hence there is no pending order for it to load and update. So I think the first approach may be the better one. But the downside of only responding to Payment Succeeded events is that if the payment failed, there will be no record of the Order existing at all, other than in the event log itself.
Hi Grygoriy Gonchar, thanks for the nice video on data consistency.
I have a question : Where is the "Application Event Log" containing the order details (Request, Payment, Flight etc) stored ? I mean in which service ?
In the example you mentioned, in "makemytrip" there are 5 services, where is the "Application Event Log" Maintained ? Is it maintained by a separate microservice (SAGA) or in one of the 5 microservices (Order, Payment, Hotel, Flight, Transfer) ?
the presenter is very well educated, please add his name for reference :)
thanks for the great talk
Great sharing video and thank for tutorial friend, nice meet you ~~^^ ^^
Thanks for this great talk
Thanks so much for this tutorial.
Awesome presentation. No questions from the audience, made me surprise!
Anyway, He mention in Kafka vs RabbitMQ, with Kafka there are no retries out of the box. How come that happen?
In Kafka, there is no easy way to retry processing a specific message without stop moving the cursor and blocking the whole partition processing. Sure you can retry processing a message easily by not moving the cursor forward, but that would delay processing the next messages in that partition. In contrast in message brokers like RabbitMQ you can do that.
What he said ^ I'll just add that different use cases have different requirements for strict ordering of event processing. In some applications, it is critical that the next message not be processed until after the current message has been processed, and hence the cursor should not be allowed to move and the whole partition's queue blocked until the issue is resolved. In other use cases, one or two retries can be attempted and then the failure recorded and the processing allowed to move on past the failed message. And in some use cases, a given event should only be processed one time and the queue continues, regardless of whether it succeeded or failed.
All of these approaches are easily achieved using Kafka's offset counter / cursor. But the developer needs to make a conscious decision about which approach is appropriate for a their use case.
Does it affect system performance?
usually consistency and availability are a compromise , so in a way yes ... more consistency is less
nice presentation
Yeah but bad audience. There is a sea of questions sailed by!
I wonder if anyone if the audience was even listening to this guy? How come you cannot have a question about this topic?
The audience asked many questions afterward in person :)
21:15 deal with incositency
Need more examples
Спасибо, коротко, по делу, понятным английским. А на русском есть то же самое или похожее?
Из моих докладов, к сожалению, нет
@@grygoriygonchar6994 Спасибо. Может, есть на украинском? :)
@@grygoriygonchar6994 Спасибо, текст доклада очень помоает ebaytech.berlin/data-consistency-in-microservices-architecture-bf99ba31636f
paid*
Too much content in too little time
I agree ... and some of the patterns were not clear segregated. For example, he talked about reconciliation first ... but later seemed to talk about CQRS etc (stuff in the SAGA pattern). Is reconciliation part of the saga pattern or is it a different pattern altogether? Was confused on this.
good content but poorly presented
This is how you not do a presentation.. a slide should illustrate the idea. Too much talk
ZZZZZZZZZzzzzzzzzzzzzzzz
difficult to understand his English. painful. sorry. Otherwise, it is good
Blockchain
No code, no believe.
😂
Too much content in too little time