Database Replication | Synchronous vs Asynchronous | System Design Tutorials | Lecture 18 | 2020
ฝัง
- เผยแพร่เมื่อ 28 ก.ค. 2024
- This is the eighteenth video in the series of System Design Primer Course. We talk about one more important component of System Design: Database Replication.
We want software engineers and aspiring software engineers to develop basics and get ready for the world of interviews as well as excelling as a Software Engineer.
------------------------------------------------------------------
Recommendations
------------------------------------------------------------------
Our full courses on youtube:
✒ System Design Primer Course: • System Design Primer C...
✒ REST APIs made easy: • REST APIs MADE EASY
Some paid courses that we recommend:
✒Educative.io: bit.ly/3qnW5ku
✒Interviewready.io: get.interviewready.io/ (Use coupon code SUDOCODE for extra discount)
------------------------------------------------------------------
About Us
------------------------------------------------------------------
Created and Instructed by:
Yogita Sharma
✒ LinkedIn - / yogita-sharma-83400b55
✒ Instagram - / sudo.code1
✒ Facebook - / sudo.code
✒ Medium - / yogita088
Post-production(editing, thumbnail etc) managed by:
CiKi
✒ Website: www.ciki.co.in
✒ LinkedIn: / 74735937
Colors and design by:
Naini Todi
✒ LinkedIn - / nainitodi
Both Arpit and Yogita are software engineers and want to help other software engineers become better by providing high quality and well researched content by adding their creativity and teaching twist.
------------------------------------------------------------------
Join Us
------------------------------------------------------------------
Hangout with sudoCode:
✒Discord Server: / discord
For business:
✒Email: sudocode.yogita@gmail.com
------------------------------------------------------------------------------------------------------------------------------------
Timestamps:
0:00 - Intro
0:53 - What does replication mean ?
5:45 - Understanding replication lag
7:21 - Replicating Synchronously
9:44 - Advantages of synchronous replication
11:24 - Asynchronous Replication
16:10 - DB replica versus Snapshot
17:27 - Summary
Hello peeps! Hope you like the video. Apologies about the traffic noise in the start. Was recording this late at night in my home town in India. :)
No problem at all.... This video was really helpful and I sincerely encourage you you to please make more videos like these.Thanks a ton!!
Hi, Firstly I would like to thank you for creating such great videos on System designs. I have just one thing to ask that I can't the reference links in your description for couple of videos. Can you please share across those links, will help me to dig deep in the topics you covered. Thanks in advance
This is great content, thank you so much. could you please do more content on db replication and faults and failure handling like the previous video in the series.
Can anyone help me how i can actually implement it, i hve understood the concept but to to implement
Thank you so much, I've watch couple of videos but you are the one that makes me get it!
I was searching all over the internet for System design videos that are basic and which I can easily understand. This series helped me a lot in understanding the basic concepts. Nicely explained. Now I'm thinking about why I didn't start searching a little early 🙂. Please add the reference links in your description. I can't find them. Thanks again Yogita.
loving this video series - finding it very helpful!
That's a great video. Would love to see a lesson on multi-master replication, and how to handle conflicts in those cases.
I was confused with aerospike db for implementing a ratelimiter algorithm to store and access atomic counters in faster way, as it has many read write polices between master and slaves. Finally this video clears my doubts to understand such systems.
Awesome one, Thanks for sharing.
Excellent Explanation. Keep it up 👍
Awesome video. Thanks you very much!!
Very good video with wonderful explanation. ✨✨
Very good videos. Thanks for the content.
couldn't be more perfect!
Thanks for the very helpful video.
Hello Yogita,
Great informative video as always but was really looking forwards to learn more about the different replication models which you were touching upon briefly throughout the video. You seem to stress a lot on exploring the additional resource links in the description of the video. But unfortunately, they seem to have been missed somehow. This has been the case in past few of the videos too.
Could you please update the descriptions of video to have those links added?
Thanks in advance.
Thank you so much it is really helpful.
thanks for knowledge sharing...😊
Superbly made ❤
I understand very easily by the way you teach. So please make a in depth video of database replication as it becomes very important when data transaction is very large.
Nice Ma'am!
Thank you so much 👍🏼🎉⭐🙏
a very good explanation.
Thanks for this !!
Neat explanation
hi, thanks for the video Yogita :)
, one doubt is whether the snapshots are also distributed across all the nodes or specific to a node
@sudoCODE, Yogita, thanks for the detailed explanation. Could you please update the links as you mentioned, couldn't find them in the description box. Thank u.
Yes please, replication primer 👍
Thank you so much!
great vid
thank you..😊😊
Thank you very much
love the video
Another great Video! :)
Thanks again!
Conceptually you are the best teacher mam...
Thanks Mridul
Nice👌🏻
Clear concept
Nice explanation!!
Also, it would be great if explain a bit about multi-leader replication, and how to handle conflicts in that case.
Please keep on making videos!!
Thanks for sharing your knowledge ma'am!!
Thanka a lot Rupesh. Yes I will definitely make video on leader election, consensus, paxos etc.
Very useful video..thanks a lot!
Glad it was helpful!
Very good video ! Thank you
Glad you liked it!
Nice video !! A replication primer for sure :)
Hehe. Thanks
Awesome content.
Thanks
hello sudoCode, as you explain very welll but still want to know more details the things, please attach some online linkes or materials for every topics from where we can get the full/details context.... Thanks In Advance :)
Greate content thanks a lot.
But I have a doubt that in all the replication methods (sync or async or hybrid) If one of the replications (write in R1, R2, R3) write fails (suppose in R3 write fails) then how to manage that situation. So in this case Data inconsistency problem might occur.
Correct me if I am wrong.
Thanks again for producing beautiful content.
Thanks
Hi Yogita, I really liked the series, especially the real-world analogies given at the start. However, I would request you to please add the resources in the description, as they are missing in many videos.
Sure 😊
Awesome content ☺️
Thanks Shani 😊
Short and sweet introduction to replication. Though you mention it is short information of big topic, in replication if primary goes down then one of secondary become primary and in case of asynchronous how to make sure that secondary selected out of consistent only.
That depends on implementation level.
Great video! Can you make a video on data models? For example Relational/Document/Graph/wide-column? Their applicability, use-cases and examples?
Guess what? I already did :) th-cam.com/video/O_c7lzNbcKo/w-d-xo.html
Big ups
You are going great job.
Please make more detailed video database replication .
Sure I will
No resources in the description
i want to know more on data replication engine,i want you to make a complete vedio for data replication project,how it works....the way you teach is greate.
thanks
Very insightful mam
Thanks a lot
Nice explanation.
Thanks
I liked your video. The explanation is simple and easy to understand but can you please add the links that you mention in your video? I cannot find them
Thanks a lot for the nice explanation. Shared the channel link to the coding community sites to acquire the greater number of audience for such an informative topics of system design. Pls do post video on system design example
- facebook feed, google serarch, instagram working , uber architecture.
Thanks
That's very kind of you! Thanks a ton.
Really good content mam
Thanks Kalyan
There could have been an overall slide for this video on strategy, like you have done for caching video
Hi Yogita, Thanks for providing such good and clear content! I was looking for the links you said (in video) to be their in description but cannot find any here. Is there someother resource i need to check other than the courses ?
+1
Hey, I am a bit confused about synchronous replication and how it solves consistency issues in case of high latency between the replicas. You explained that the write is initiated on the master and is broadcasted to all the replicas. The replicas execute the write command and send back acknowledgments. What happens if there is a read during this process? It is quite possible that a read request on the master is going to be different from the read request to one of the replicas. Do we accept that inconsistency? In a write and read-heavy system, this will happen a non-trivial number of times.
i think till the master sends acknowledgement to the writer service for synchronised replication for example updating a order, other services like for example invoice service won't enquire about that particular transaction even if it accessing the inconsistent replicas. They might want to enquire about already consistent transactions. Hope this makes sense may be..
Please explain about eventually consistent
Thanks for this amazing video ❤ 🙏 i have a doubt when I insert bulk data in million then slave started 100% cpu utilisation and due to the same my server -read stop working till the operation completed. Pls suggest how I can handle this situation?
Thanks in advance 🙏
hi really great explanation, it will be nice if you add subtitle for people like me who rly hard to understand spoken English (I'm not native English speaker), it'll really help me, thank you in advance :)
Hi Yogita, while we do understand the different methods to update the replicas , be it hybrid , sync or async , in case of a transaction driven app , as you said , can be banking , which of these has to be used since I see a glitch in all three in terms of data consistency for example Hybrid - R1 is updated and main DB says write complete , user is accessing R2 which means user wouldn’t get the updated data .
So isn’t it like it should first update the replica which user is accessing and all others later ?
Hi.. I could not find the link to the docs u mentioned? Could u pls share that
Hi, I like the video I am unable to find resources in the description.
so can you please tell me more about table replica
Hi Yogita, thanks for sharing these wonderful videos. it is really helping us to understand the concepts. One Question though - in every video you mentioned the link for additional content but i never found it. Generally it has details about past and future video, link about your linkedin and hangout... i am not sure if i am looking at the right place. Please suggest.
Hey I am getting same problem . Have you found it
@@AnkurKumar-vx3xv no
how all replica DBs will be locked for read operation when master is updating using synchronous approach ? that time also read can happen and reader can get wrong data if update is not completed from master. ?
U forgot to add or mention about the latency topic in any video. Pls check and make a video on that as well.
I don't know but whenever I see your explanation it feels like my big sister is teaching me.. Although I don't have one 😐... Amazing content can't imagine it's for free..
This made my day 💙
Hello maam, i have a doubt in the method of solving replication lag. The lag can happen between the secondary DBs as well. The acknowledgment will also take time to reach and will the time taken i suppose will be exactly equal to the the lag. Hence, we were never able to solve the problem
what is row based replication & statement based replication ? Please explain.
What will happen if Replica fail to update? What will be the best approach to handle this ?
Streaming is like broadcasting all database in cloud and accessing is available all time as soon as new data uploaded
Hi Yogita, nice videos very informative - keep it up!
I don't see links to additional resources in description from past few videos - Can you please check it once? I would love to read/explore more on these topics using the links you have in mind. Thanks :)
Will check and try to update.
@@sudocode Hi Yogita, do you have the additional resources? In many videos, they are not there in the description. Thank you for the great content!
Nice! So database system will take care of copying data to slaves, it is matter of configuration on database system level....what about routing read n writing request to the respective db node,would that b implemented on the application level via some gateway to filter n send to respective db i.e., read or write???eagerly curious :-))
Excellent !! I dont see any references or links in the description as you mentioned...
let me check the links.
I'm thinking of created an application service that will act as a db read cache. But pass thought writes.
If we also able to demo this theories
I am trying to build demos.
I'm confused between this and CQRS .
In synchronous replication only one of replica is in sync with primary db, correct me if wrong.
It depends on the implementation. All replicas can be first synced before committing writes.
Mam you always say "links in description" for further read. I couldn't find anything in the description.
Please try to create a database replication course.
Can’t promise. Will do if there’s a lot of ask for it.
Ma'am please share notes of these lecture.
Will try!
the way you explained synchronous replication feels incomplete. To me, if the replicas are serving the reads (as described a minute before.) then there is a time when some of the replicas have done the write and sent their ack, and others have replicas have not yet done so. At that point the replicas which have not yet acknowledged will be reporting the old value, and those that have acknowledged should return the new one.
I suspect there is another round of of packets to circulate to confirm (commit) that the write has succeeded, and only after the second round trip should the master answer that the write has succeeded.
my notes below:
Replication:
Pros - Fault tolerance, secondary mains, latency reduction, cqrs, Performance improvement, availability.
Cons - dissynchronization(replication lag) -> Data inconsistency.
Solution for data consistency
Read after write consistency
To be continued....
#SR7
I can hear the traffic in the background xD
Hey Yogita, Can you please explain what happens in case of synchronous replication where there is data exchange from both sides at the same time?
example: data variable x is updated to 1 at time t1 on primary DB, and at the same time t1,data variable x is updated to 2 on secondary DB. How does DB handle these situations? do database engineers keep a check on race conditions while configuring the replica setup?
This is a classic concurrent event case. In distributed systems there is no way to conclude the exact ordering of concurrent events, so it depends on the consensus algorithm used to reach and agree on a result.
hmm the location of Ireland seems a bit off..
Yeah. Sorry for that Van 😑
where is link , are you getting lazy
Master slave is a wrong terminology, leader worker is the right one
sorry no hates but she looks like Github LOGO
Thank you it’s an honour 😃
hello sudoCode, as you explain very welll but still want to know more details the things, please attach some online linkes or materials for every topics from where we can get the full/details context.... Thanks In Advance :)