6. Consistent Hashing in Hindi with Example | System Design - Consistent Hashing | High Level design
ฝัง
- เผยแพร่เมื่อ 2 ก.ย. 2022
- Consistent Hashing is a distributed hashing scheme that operates independently of the number of servers or objects in a distributed hash table by assigning them a position on an abstract circle, or hash ring. This allows servers and objects to scale without affecting the overall system.
So main purpose is to distribute the load among the servers equally and during addition or deletion of server, minimum number of keys to be rebalanced.
prerequisite: • 5. Scale from ZERO to ...
Join this channel to get access to perks:
/ @conceptandcoding
Telegram group: t.me/+KwJhmit5yDczNzY1
#consistenthashing #loadbalancing #systemdesign
have refered many videos on consistent hashing but this video cleared all my doubts and crisp and clear content, well narrated 👏
thanks
@@ConceptandCoding I have a rudimentary question here. Consistent hashing will be used for scaling up and scaling down systems right ? for any unexpected server fail over, we need to relay the redundant systems running at separate server/cluster right ?
best video on consistent hashing
Thanks for making consistent hashing so easy to understand, especially where you explain about Virtual Nodes.👍👍
Glad it was helpful!
Recently i started watching your videos, they are very helpful, thanks for making these videos ,
thank you
Easy to understand, Thanks
Thanks, for this crisp explanation . I
First time watching you, and it's really great experience, your way to teaching is just incredible 🔥.
thank you
Very very Clear explanation🔥🔥
Glad it was helpful!
Hare Krishna..! Bhaiye
Great work
Radhe radhe. Thank you
Thanks for your efforts buddy.keep up the good up work like these.
thank you
In case of virtual objects, when a server is taken down. All the associated virtual objects will also be removed from the ring. In this case the rebalancing will be high.
Actually, not; if there are n keys and m servers, then on average n/m keys are associated with the server; and virtual server help to go close to this average in uneven distribution.
When the main server is taken down we can choose some virtual server as the main one based on some algorithm since the virtual server is a replica of the main server, what do u guys think?
The virtual servers are taken out from different locations in the ring. The concept of consistent hashing with virtual servers is to distribute even load to all the servers.
aapki videos bohot helpful rehti h. accenture innovation chalenge pr bhi ek video bana dijiye please. Hame thodi help ho jaegi
Will we be able to just rebalance 1/n keys even if a server which is replicated thrice goes down?(In virtual objects) and what if the number of servers exceeds the size of the virtual ring?
Great explanation sir!! Keep up the good work!!!
Thanks a lot!
Owo❤. Thanks 👍
Amazing content, crystal clear explanation.
Thanks
@@ConceptandCoding One small question,
I too have an iPad but looking for good tool to make notes and diagrams. Could you tell which software you used in this video to present these diagrams.
@@ayush_singh6203 one note
I have a question, we are figuring out the initial placement (lets say each DB server) using a hash function. After that, we place the replica of those db servers (virtual servers) to balance the storage of any keys (data). Now, if we get a read request, even after doing a hash (and mod) of it, we will have to know the locations where the virtual servers are placed as well to know which server that read request should be mapped to. Will we store the mapping of virtual db server and its location somewhere?
I think the mapping of virtual db and its location can be stored in a centralized metadata store, such as a distributed key-value store or a metadata database. So when a read request comes in, you would first hash the key to determine the initial server where the data is located. Then, you would consult the mapping of virtual db to identify the actual db where the data replica resides. Finally, you route the read request to that server to fetch the data.
Great explanation.
Pls also make some mock interview on machine coding round.
sure
instead of randomly assigning positions, we can implement an algorithm, which will assign positions to all the servers , such that all the servers will be equidistant from each other. If all servers will be equidistant from each other Like they were in the early examples of Consistent hashing, then I think our purpose of implementing consistent hashing will be fulfilled.
feedbacks are welcomed
Right
If you plan to keep all the servers equidistant from each other, then an addition or a removal of a server will lead to movement of servers on the ring which would lead to more rebalancing. No?
@KumarSaurabh-nu9cr Correct
would like to see live demo of this to understand it better
@ConceptandCoding Thank you for touching upon the "servers being equidistant" and "servers being next to each other" parts in the ring. I have a very lame question. When we create virtual servers, to solve the issue with the servers being next to each other, is there any logic we follow, like a math function that generates random points in the circle? How do we know that we have achieved 1/n% of rebalancing in this case? Can you give one small use case where we can understand how to calculate the 1/n% of rebalancing after creating virtual points?
Like I said, apologies for my lame question. I would really appreciate it if you cleared this doubt. Thank you in advance!
If hashing function provided to partition key makes node assigned random. Then why it is advised to choose a partition key as random as possible
Absolutely worth it. Can you also share the reference or book where you prepare the material?
Generally i read, "Design data intensive application"
But this topic i read from DynamoDB research paper, as DynamoDB also uses Consistent Hashing, so they explained it in detailed.
Nice explanation
Thanks
I joined membership yet i am not able to access this content. Please help.
when we talk about to replicate the servers of virtual objects , are they actual physical replication of machine or just virtual replication ? and whatever it is but how we come up with the location on the Ring to replicate them ?(virtual object) because we already used mod hashing to place the actual servers on the ring, and what about hash colison while calcuating hash values for Servers, ? two or more servers can have same hash-code so how to handle it!!?
Really good questions, But since I have seen consistent hashing at multiple places here are the answers:
1. It is just a virtual replication not the actual replication. We could do it by having multiple hash functions per server. The hashing functions should be in a way that satisfies the properties of keeping them in a ring.
2. hash collisions interesting point, but since we are just assigning the load, suppose k1 and kn colloide they simply move to the same server.
For HLD interview do they ask us to write the code or just to explain the concepts ? because I never saw any youtuber explaining using code
foe HLD no coding is required only concepts and for LLD you need coding
Great Explanation🎉
Thank you
Great Explanation!
i have few doubts...
let's say a key is requested repeatedly and that key lies is one of the replica's, so request goes to the same replica everytime.
how's consistent hashing helping here?
aren't we putting heavy load on the same replica every time?
how's the traffic equally distributed in this case?
(1) A consistent hashing helps even distribution of unique keys across all servers.
(2) A single key request is designed to hit one unique server where the value resides, so if one request arrives repeately for million times then by design it should hit the same server million times.
(3) Bigger question is what situation made this usecase possible? and
(4) Do you think caching those request/respose could help in this usecase?
@@amituchat4865 Or we can add a Rate Limiter
how are we deciding the size of queue in virtual ring ?
suppose we have flood of request with key (k1) , in this case all request will go at server 1 only , so how does consistent hashing is solving the problem ?
Ack of the question
that means your hash function is not proper. or may be you not choose right colume for hashing. because even distribution is responsibility of hash function.
In our example we have taken the virtual ring size of 12, but how do we decide that number? Is it the maximum number of nodes possible we assumed?
Could you please tell us.
It's just a example i took may be in reality for big companies it might be in 100a or thousands
@@ConceptandCoding thank you so much 🙏
We are designing insta and sharding(horizontally) based on USERID, definitely we will fall into the problem of hot users and ovio shard will grow big as ronaldo posts a lot.
How do we handle such case ?
Note: Sharding by POSTID is very oldschool. Querying db for getting ALL post of a user will be an expensive operation which seems one of a primary ask from our system, here.
Thats why Consistent Hashing technique is used, as sharding by postid is old-school :)
Whatever problem you have mentioned it can be resolved using consistent Hashing itself.
@@ConceptandCoding Can you please elaborate how?
Sharding based on userID will put one of the hot users in a single shard. If that goes down, user's data goes down. How does consistent hashing solves our issue here? It will be used to allocate a userID to a specific server only. Correct me if I'm wrong. Even if we have virtual replicas, still user data will go to a single shard. How do we develop a robust solution here?
@@piyushmathpal4244 i think the hash key of diff post id wont go to the same server, it will be distributed over circle and can be handled accordingly by diff nodes
thanks....bhai notes bhi dediya kro
If we have 100 keys and 5 nodes then how many rebalancing nodes are needed in total? I didn't understood that point.
ACK of the question,will reply by tomm.
How to know total no. of keys ?
Hi, what would happen of the no if servers reached to the size of virtual ring .?
Hi Fazil, as per my understanding, the hash space(total no of nodes it can have) for Consistent hashing is Huge (generally in 2^256) so scale up issue do not come here.
@@ConceptandCoding Thanks a lot for clarifying.
Bhai 1 raise to n nahi 1 by n number hoga vo
What does hashing the server signify at 18:48?
Passing the name or ip address of the sever through a hash function
Suppose S3 is getting request of 90 keys. Then if we remove s3. Then all the requests going to s3 needs rebalancing similar to normal hashing. Isn’t it?
Only benefit of consistent hashing is even distribution at particular time using virtual locations of every servers? And not rebalancing ?
Why we are taking 12 sectors in the virtual ring ?
Hi Shubham, i just took an example, generally its a large number not 12.
@@ConceptandCoding how to decide, how many we need to consider for the virtual ring?
@@subhamacharya4472 Shubham it depend upon the company needs.
And how can you decide ring sectors as requests can be in millions and ever increasing
What if I don't know hindi.. And want to learn the content you are creating
Subtitles??
There are courses in other languages too. If there is none, create one. It will increase diversity of that language
1 raise to power n keys nhi ho rha rebalance.. Please yaar, jo likh rhe ho woh toh bolo kam se kam
Comrade Shreyansh, you have been drafted. Report to your nearest conscription office where you will be tasked with building scalable military-grade systems which would be invulnerable to Ukrainian and NATO cyberattacks
Military mainly uses real time systems I don't think scaling them largely is super important untill unless you are trying to take over whole world at once 💀💀
Please speak in english or atleast add english subtitles
Hi Ajay, i used mixed language, it would be easily understand
@@ConceptandCodingAs a non hindi person, any Hindi word goes above my head,making it impossible to comphrehend the lecture. Can you please at least add english subtitles ?
@@ajaysabarish9645 I understand your concern, if you do not have option like subtitles etc then just listen attentively, most hindi words used are filler words (one time learning, by making a list) but the key words vocab are always english. In the quest of making hindi words list, you will end up listening attentively and hopefully get a crystal clear understanding of a topic as a biproduct. Hope it helps
Not able to join telegram group via given link. Please help @Concepts && Coding
t.me/+KwJhmit5yDczNzY1
Pls try this
Bhaiya hld ka members only free krdo 🥲
Hi shrayansh, i needed some preparation guidance from you. Can i connect to you over LinkedIn or Facebook whichever is comfortable for you?
yes linkedin i am good.
I sent you connect request. Please accept it so that i can message and get some valuable guidance from you. My name is shashank Kumar
My name is shashank Kumar
Hi i have one doubt so let’s suppose if two servers not replicas point to same hash key which means let’s suppose you have 3 servers and 2 of them points to same hash id which means 2
Then how and which server will take that request??
You are saying two servers will have same hash key, in general this should not happen, because we always try to handle hash collision in these designs.
But still there are some chances that servers can have same I'd, in this case you can consider other parameters to route the request like priority, session count, etc. (Depends on the use case, but this is very very rare)