Key based Sharding | Shard Key | Hash function | Advantages and disadvantages | 2021
ฝัง
- เผยแพร่เมื่อ 27 ก.ค. 2024
- This is the twenty-second video in the series of System Design Primer Course. We talk about one more important component of System Design: Key based sharding.
We want software engineers and aspiring software engineers to develop basics and get ready for the world of interviews as well as excelling as a Software Engineer.
More on the video:
www.citusdata.com/blog/2018/0...
blog.yugabyte.com/four-data-s...
------------------------------------------------------------------
Recommendations
------------------------------------------------------------------
Our full courses on youtube:
✒ System Design Primer Course: • System Design Primer C...
✒ REST APIs made easy: • REST APIs MADE EASY
Some paid courses that we recommend:
✒Educative.io: bit.ly/3qnW5ku
✒Interviewready.io: get.interviewready.io/ (Use coupon code SUDOCODE for extra discount)
------------------------------------------------------------------
About Us
------------------------------------------------------------------
Created and Instructed by:
Yogita Sharma
✒ LinkedIn - / yogita-sharma-83400b55
✒ Instagram - / sudo.code1
✒ Facebook - / sudo.code
✒ Medium - / yogita088
Post-production(editing, thumbnail etc) managed by:
CiKi
✒ Website: www.ciki.co.in
✒ LinkedIn: / 74735937
Colors and design by:
Naini Todi
✒ LinkedIn - / nainitodi
Both Arpit and Yogita are software engineers and want to help other software engineers become better by providing high quality and well researched content by adding their creativity and teaching twist.
------------------------------------------------------------------
Join Us
------------------------------------------------------------------
Hangout with sudoCode:
✒Discord Server: / discord
For business:
✒Email: sudocode.yogita@gmail.com
------------------------------------------------------------------------------------------------------------------------------------
Timestamps:
0:00 - Intro
0:27 - What is key based sharding ?
1:35 - Shard key and Hash function
3:44 - Advantages of key based sharding
4:37 - Disadvantages of key based sharding
5:44 - How to use a sharding key ?
7:20 - Outro
Watched lot of videos but your one is the best. Love you. Thanks.
A concise and to the point video. Thank you
Good to have very informative and clear understanding video from Indian
Keep it up and wish for your great journey 🙏
simple and very informative
Hi Yogita, Your videos are very helpful for understanding system design concepts throughly. Can you please make some videos on Cassandra and Hbase too? More specifically on Data Modeling, and using some existing system designs examples of where can we use MySql, Cassandra, and Hbase.
Thanks for your wonderful contribution to our preparation.
It is very Informative Thank you
It's great source to learn system designing..mam can you please make videos on interview design questions
Thank for nicely explanation
wonderful @sudoCode chossing a right Sharding key or combinations is crucial and needs advised an deep decisions ... kudos to you!
good work..
very informative Yogita, thank you for making this video. One question what will happen if we have to move multiple tables into different physical sharding? in that case fetching the data requires to contact multiple shards, right?
Thank you very much
Wow it's so informative video 😊
Thanks a lot 😊
very informative yogita, thanks a lot!
I have a question, say Tinder supports location based sharding although if I go to another city, I will have to be immediately added to that DB, and if we consider multiple such movements, how would tinder handle such dynamic changes?
thanks for the video. I have a question, what the different with partition key? looks like same ya for the function after all. Thanks
Very good introduction to sharding... How is replication taken care of in case of sharded DB servers.. How is it made fault tolerant..
Nice lecture Mam😇😇
Keep it up.✌✌
Thanks a lot
One disadvantage is that in such sharing range queries can’t be performed. For example list top 10 employees based on their salary.
Please if u can create a video on how we do mysql sharding from scratch by practical example. Appreciate ur effort. Great video content. Keep continue 👍
Noted. It is in pipeline.
Rather than moving data to shard three cant we modify hash function in such a way that every write operation will store to shard three?
Please also consider implementing these concepts
will do
Is the right shard key _id :hashed ? If not explain what is the reason
I have a confusion on where load balancer for DB servers come into picture if DB is sharded.. I am not able to visualize Load balancer + sharding DB servers.. DB sharding is done to distribute data.. then we would have different DB servers.. but load balancer cannot send request based on the load to any of the sharded DB server instances? Kindly shed some light on this
Hi Yogita, thanks for the video, very nicely explained. One doubt...
Since the records are distributed in shards based on the sharding key(say city), how will each shard have equal number of records?
In case of city Name as a sharding key, There is not any guarantee of equal no. of records in each shard,
Short, clear, and easy to understand!!
Ma'am when we will use the city as shard-key then let's say a user is in berlin on 20 Feb 2021 and now on 24th Feb 2021 he moved to Banglore, so in that case how the situation would be handled because earlier the write operating was going to key = Hash(berlin) shard but now the key will be changed as key = Hash(Banglore). So if that user sends a read or write operation then the data is not there in the shard having Key = Hash(Banglore) ??
Exactly, that's why in key based sharding we choose keys which are static and don;t change frequently. If a user's city is changing frequently then we don't choose that as sharding key
weather it can be the case that out data is not equally distributed among all the shard's and there is large load on one of the shard's....weather there is a way to over come this? sorry if my ques is bogus...
Nice Video. Suppose we have 2 tables in the database. Should both the tables be sharded if we go for sharding? If so how can we derive the sharding key in multiple tables cases?
Depends on table sizes. Both shall have different sharding keys depending on the use case.
Also please cover topics such as :
- Consistent Hashing
- Quad Tree
- Web Socket , HTTP and LONG POLLING over HTTPS
- REDIS CACHE
- CASSABDRA
Thanks
All of them are pipeline!
@@sudocode any update ? I am specifically looking for consistent hashing
Quick question: For systems like Netflix - if we choose a sharding key based on the city and the user is travelling to a different city - how does the system handle this? What are the strategues that we could use?
Mostly they key won’t be chosen as per city. Key is chosen in a way such that data can be equally distributed. In case of Netflix data could be divided on the basis of categories, origin country or even alphabetically or creation date ranges etc.
Nice video! One question here, I am kind of wondering how to query on the sharded database? Will the query run on each of them?
Nope, the algorithmic sharding logic will take care of routing the query. This is one of the advantages of sharding that the read and write query performance increases a lot since these queries run only on relevant shard and not the whole data.
@@sudocode If the shard key is user's first name, but the query selecting all the users that have the same last name, for example, Smith. In that case, it has to query through all the shards, is that correct?
@@streammxc Yes, in that case you have to send the query to all the shards and aggregate the result. That is called scatter and gather.
And this is one bottleneck in key based sharding.. any non key based read query will not be efficient.
Thanks a lot for such a detailed and informative video. Request you to upload some video on full system design on UBER, FACEBOOK or any other system so that we can apply all the concepts that we have learnt into one system. Thanks a lot for your time and hard work. Truly appreciate your dedication. Kind regards.
Yes, will upload. :)
How do you handle changes in shard key? For instance, the record is stored in DB1. What will happen if the shard key changes for the record and say the hash function will return DB2? The hash function will return a different database and the record will not exist there. How do you handle this scenario?
Usually you don’t change shrad key much often. Let’s say you having shrad key on City Coulmn and persons are mapped to shards based on the city. Let’s say a person living in Newyork mapped to shrad 1 and he later moved to SFO, in this scenario, any read or write request will be passing through partition awared load balancer , which would first calculate the hash of that shrad 🔑 which is 1, and write updates the city column which is now SFO. Now any CDC I.e Change of Data Change will detect and reshrad this key to shard 2 ( SFO) associated and remember this shrad 2 may be replicated for better throughout so, we need to make sure we created this entry in all replicas.
In this Key based Sharding we are first running the hash function on userId and based on the response we are adding data to Shards? In that case if I have 5000 rows I need to run hash function on 5000 userIds and then decide in which shard it should store?
yep
👍
suppose the data is shared with location as key and now i want to fetch the information using the user Id and dont know the location. So, how could i find out in which shard the user with given user id exit ??
need to aggregate all the shards data and then do a filter on them
I am not understand how to shard key choose, briefly explain in the comment box
apart from good content, nice decorated heading is awesome:-)
Yeah thanks
shard key 1:15
hash shard is known as algorithmic sharding 2:00
disadvantage hash shard 4:50
to solve the hash shard problem use consistent hashing