Head to database.husseinnasser.com for a discount coupon to my Introduction to Database Engineering course. Link redirects to udemy with coupon applied.
As a student, for my project, I had to deploy my application on multiple servers for scaling (horizontal scaling). Since I was using docker and docker swarm to help in containers and orchestrestration, came the time to think 🤔 on how to handle the database. Question was : How do I horizontally scale the database while keeping its state across these servers? Ans : Distributed Transactions. After going deep in it and understanding the level of complexity behind scaling a database horizontally, my prof advised me to just use a single database, cuz this problem is still, as of date not yet completely solved. Distributed Databases is still an active area of research and development. He/She who solves the problem is set to impact the industry massively. Just food for thought and Thanks for the video. :)
It depends on your system, I don't believe absolute consistency exist in distributed databases. But eventual consistency is pretty much solved and used in any large scale system.
We can maintain two leaders, both of them accessing some database like etcd which is an inmemory distributed key value store. So if one goes down, the other is there. Or we can just keep one and spin up another when one goes down.
Upon revisiting the video a year later, I discovered that Raft implements a 2-phase commit protocol. If the leader fails, a random leader election occurs with the highest term index.
We could've done other way right ? Like as soon as co ordinator realized that he didn't got response from one DB , It will ask other DB's to rollback that commit and don't accept Tx's anymore until that offline node comes back online and then all of them will commit tx. But ofcourse , As CAP theorem says we are trading consistensy for availability then.
Hussein you're a GOD, you literally published this video 2h before my exam and now I've understood that! Thank you very much! Hope to see more videos about DB stuff 😉
I have a small doubt regarding the Three-Phase Commit (3PC) protocol: If the coordinator fails after sending the 'CanCommit' requests in 3PC, how is this scenario effectively managed differently from the Two-Phase Commit (2PC) protocol, where a similar failure leads to a blocking issue?
Is it moments like these when I should start considering NoSql for microservices rather than trying to achieve a fake sense of acidity through XA and Saga implementations of DT?
It is not necessary to synchronize the distributed databases. There are two arguements for this. 1. All human are distributed, our memories are not synchronized. What is important is just network communications to pass messages. 2. If you synchronize the databases, it is the same as having one database. This is wasteful and should be avoided. Only minimal backup as 'disk' is necessary.
Head to database.husseinnasser.com for a discount coupon to my Introduction to Database Engineering course. Link redirects to udemy with coupon applied.
Please talk about Saga Design Pattern in the BE Show.
What is BE show
@@arenazo the better question - what is sega design pattern
@@filcondrat it's about coordination of microservices
@@JohnZakaria I was somewhat ironic
I heard MAGA.
As a student, for my project, I had to deploy my application on multiple servers for scaling (horizontal scaling). Since I was using docker and docker swarm to help in containers and orchestrestration, came the time to think 🤔 on how to handle the database.
Question was : How do I horizontally scale the database while keeping its state across these servers?
Ans : Distributed Transactions.
After going deep in it and understanding the level of complexity behind scaling a database horizontally, my prof advised me to just use a single database, cuz this problem is still, as of date not yet completely solved.
Distributed Databases is still an active area of research and development. He/She who solves the problem is set to impact the industry massively.
Just food for thought and Thanks for the video. :)
It depends on your system, I don't believe absolute consistency exist in distributed databases. But eventual consistency is pretty much solved and used in any large scale system.
horizontal scaling of databases means? creating a new database when the existing database is full?
Great Video!
Can you please discuss about Saga Implementation details.
We try to solve one issue and get a new one. 😅
Great video though even though I am not a dev.
Thanks for informative video . Would love to hear from you about 3 -Phase and SAGA .
2 phase, 3 phase and saga differences would be great. how does it change consistency etc.
Very good video, very well explained, Thnx !
We can maintain two leaders, both of them accessing some database like etcd which is an inmemory distributed key value store. So if one goes down, the other is there. Or we can just keep one and spin up another when one goes down.
Upon revisiting the video a year later, I discovered that Raft implements a 2-phase commit protocol. If the leader fails, a random leader election occurs with the highest term index.
Very well explained!
We mostly use Paxos or Raft in distributed databases these days.
We could've done other way right ? Like as soon as co ordinator realized that he didn't got response from one DB , It will ask other DB's to rollback that commit and don't accept Tx's anymore until that offline node comes back online and then all of them will commit tx. But ofcourse , As CAP theorem says we are trading consistensy for availability then.
Maybe you could tackle transactional outbox next? I think you could apply the same even though it's meant for databae+messaging.
We used two phased commits
Go Saga next
who wants book shelf tour
Hussein you're a GOD, you literally published this video 2h before my exam and now I've understood that! Thank you very much! Hope to see more videos about DB stuff 😉
I have a small doubt regarding the Three-Phase Commit (3PC) protocol: If the coordinator fails after sending the 'CanCommit' requests in 3PC, how is this scenario effectively managed differently from the Two-Phase Commit (2PC) protocol, where a similar failure leads to a blocking issue?
Is it moments like these when I should start considering NoSql for microservices rather than trying to achieve a fake sense of acidity through XA and Saga implementations of DT?
Database 3 : Ohh what happened, O was in a coma 😂😂😂
Make a video on mnesia database from erlang ecosystem
Ladees vashrum
Feeling , I am able to understand arabic language
It is not necessary to synchronize the distributed databases.
There are two arguements for this.
1. All human are distributed, our memories are not synchronized. What is important is just network communications to pass messages.
2. If you synchronize the databases, it is the same as having one database. This is wasteful and should be avoided. Only minimal backup as 'disk' is necessary.
First
I am hurt!🤣
🥇
Just want to ask what about timestamp protocol compared to 2 phase protocol couldnt understand their respective drawbacks🥲