If EXEC call for Txn9 comes before the EXEC call of Txn1, let's say this happens for both the shards, it "waits" for sometime for EXEC call for Txn1 to come in and finish its work (finish for EXEC + FIN) to maintain serializability and ordering. Great, but now, 2 questions - Where (as in memory or data structure or which place) did the EXEC call for Txn9 wait and secondly, how does the system know to start the pending EXECs that waited till now that are now ready to be processed? Is it on EXEC+FIN of the head of the local queue? i) Interested in data structure to know more on how and where timeouts for an operation are implemented ii) Interested in knowing how this "waiting" is achieved and pause-resume of those operations achieved in an optimised way, if it isn't something fancy if this is something that would be covered in subsequent videos, please share
I have a basic question. First the coord fiber sends a schedule message to shard thread. Then the scheduling algorithm rearranges the transactions. And then the coord fiber again sends another request message for execution? Is this my correct understanding? If yes: - Why does coord fiber needs to send exec message? Cant the scheduling algorithm put the message in the exec queue directly? - how does coord fiber know when to send exec message after receiving acknowledgment from previously sent message for scheduling a transaction?
Thanks Arpit for this video. Transactions in local queue are already ordered by transactionIds, then in which scenario scheduler needs to change this order ?
The shared nothing architecture ensures each shard is mutually exclusive. Meaning there is only 1 copy of each shard which is handled by a thread. Transaction queues are also maintained per shard not global. So how does the problem of serializability arise if there is only one copy residing in one partition/shard? Serializability would become a concern when there are replicas in the picture for handling reads which can have inconsistent states. Or am i missing some scenario?
You can have a TXN that involves keys that are owned by multiple shards. So that is where you need a coordinator fiber that is dealing with different shard threads and greeting things done. But while this is happening, because Dragonfly is multi-threaded other TXN can happen on other IO Fiber which now requires to modify the same set (or subset) of keys. That is where we need to ensure serializability otherwise the data will be in inconsistent state.
Checkout DragonflyDB - www.dragonflydb.io/
DragpnflyDB Source Code - github.com/dragonflydb/dragonfly
If EXEC call for Txn9 comes before the EXEC call of Txn1, let's say this happens for both the shards, it "waits" for sometime for EXEC call for Txn1 to come in and finish its work (finish for EXEC + FIN) to maintain serializability and ordering. Great, but now, 2 questions - Where (as in memory or data structure or which place) did the EXEC call for Txn9 wait and secondly, how does the system know to start the pending EXECs that waited till now that are now ready to be processed? Is it on EXEC+FIN of the head of the local queue?
i) Interested in data structure to know more on how and where timeouts for an operation are implemented
ii) Interested in knowing how this "waiting" is achieved and pause-resume of those operations achieved in an optimised way, if it isn't something fancy
if this is something that would be covered in subsequent videos, please share
I have a basic question. First the coord fiber sends a schedule message to shard thread. Then the scheduling algorithm rearranges the transactions. And then the coord fiber again sends another request message for execution? Is this my correct understanding? If yes:
- Why does coord fiber needs to send exec message? Cant the scheduling algorithm put the message in the exec queue directly?
- how does coord fiber know when to send exec message after receiving acknowledgment from previously sent message for scheduling a transaction?
Thanks Arpit for this video.
Transactions in local queue are already ordered by transactionIds, then in which scenario scheduler needs to change this order ?
When they reach out of order.
The shared nothing architecture ensures each shard is mutually exclusive. Meaning there is only 1 copy of each shard which is handled by a thread. Transaction queues are also maintained per shard not global. So how does the problem of serializability arise if there is only one copy residing in one partition/shard? Serializability would become a concern when there are replicas in the picture for handling reads which can have inconsistent states. Or am i missing some scenario?
You can have a TXN that involves keys that are owned by multiple shards. So that is where you need a coordinator fiber that is dealing with different shard threads and greeting things done.
But while this is happening, because Dragonfly is multi-threaded other TXN can happen on other IO Fiber which now requires to modify the same set (or subset) of keys. That is where we need to ensure serializability otherwise the data will be in inconsistent state.
@@AsliEngineering oh yes, dragonflydb is multi threaded
Now it makes sense
Thanks
Happy guru purnima Arpit 🎉😊
Does Coordinator fibre sends the next call like execute call as soon as it receives the acknowledgement of the last call like for schedule
I think this explaination needs to be more visual
Maybe I missed it, can someone help understand what was the significance of having a counter per key (in intent lock)?
Is it signify that there is a non-empty txn queue?
First comment ❤