What is Database Sharding?
ฝัง
- เผยแพร่เมื่อ 5 ก.ค. 2024
- Database sharding is a technique used to optimize database performance at scale. It relies on separating data into logical chunks so that they can be separated and queried independently. Learn more about sharding in this video.
Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! courses.beabetterdev.com/cour...
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driver Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
📚 References:
Great Sharding Article - aws.amazon.com/blogs/database...
#Databases
#Sharding
#SoftwareEngineer
honestly might be the most complete and thorough explanation of sharding.
Thanks so much for your kind words!
You've simplified your explanation like google engineers do when they give lectures, I'm sorry if that sounds strange but I've realized that the people who simplify complex things they really know what they are doing awesome man Cheers.
Thank you so much for the kind words!
Nice words
I am burning through all your videos. You are making me a better SAAS Test Engineer! Keep up this great work!
This is seriously such a great video man. I spent the entire Sunday understanding Sharding. Not that I didn't get started with the concept, however, this video just made everything clear at the end of the day. Thank You.
Dude you make some really awesome content. Please please keep making videos! I love the clarity of your speech, voice, and presentation. I understand and can follow along in your videos a lot better than more other channels. Earned my subscription and likes! Keep killing it homie!
Thank you so much for your kind words and welcome to the channel!
Watched countless videos and barely understood the concept. Your video on the other hand explained everything along with pros and cons super simply. Thanks a ton.
Finally found some decent content over this topic. I already had an idea on this topic just wanted to revise it. Thanks a lot for making the insightful videos.
Network guy trying to get an understanding in a different field. That's an outstanding walk-through and very much appreciated. Thank you for your work and quality presentation.
Glad it was helpful!
Best lesson about database scalability I found, so easy to understand.
Great video, especially your description about the non-uniformity problem.
Thanks Rotary Dialer! Yea the non-uniformity issue is one I've been personally bitten by in the past. Glad you enjoyed the video!
Good stuff man. I love the clarity you bring to a subject. Subscribed.
Daniel, no words.. looking at your playlists content and videos …amazing. Great great effort to help people. Kudos to you 👏👏👌👌👌
You're very welcome!
Great video! Such a clear explanation of how database sharding works.
Dude, this was outstanding! Super helpful and covered everything I needed to know!
Best of the best explantion I kinda start to understand the Sharding in just video.Thank You and I subbed!
Hands down! the best explanation I've seen on database sharding, excellent!
You're so welcome. Glad you enjoyed.
Very clear. One of the best tutorial I have ever seen
New here. Loved your talk! Your presentation and teaching is elegant and simple.
Really appreciate it, thank you!
You are so welcome!
Hey dude, you're a star! Very clear and upto the point! I cant thank you enough.
Thanks for the videos. Great explaination.
you are so good at explaining concepts
Great content man!! It helped me a lot!! Keep up with the good work!
Thank you!
This was awesome. Thanks!
In most ~20min videos, I get tired soon and close them after 5min. I can’t believe your video is so good that I totally forgot time and finish watching all of it
Thank you so much Jingyi! Its these kinds of comments that keep me motivated to make more content :)
Stay safe
Daniel
Amazing explanation, loved it. Thank you, it will help for the future interviews I have.
Glad I could help!
Great explanation, Daniel. Thank you
You're very welcome Anton!
Best video ever made on sharding
Great explanations! Thanks, Keep it coming!
Thanks Sharon!
Very well explained, great video!
woww...!! great videos, great presentation, great explanation. thank you, keep sharing..
Super clear. Thank you!
incredible explanation, thank you!
best explanation of sharding i've heard!
Thank you very much!
Perfect explanation. Thank you
Excellent presentation, very good explanation 👍👍
Bro I'll watch anything you make. If you made a video teaching me how to watch paint dry I'd take notes. Keep up the damn good work my mans.
Great job. Very well explained!!!
Thanks so much Jackson! Glad you enjoyed :)
Had a hard time grasping on what database sharding actually meant but your video really helped me understand it, thanks! :)
You're very welcome!
Thanks for the straight forward easy to grasp concept of sharding. Give this to someone else and we would have gotten a bunch of technical wordy mumbo-jumbo.
This is great and super clear. Thank you!
You're very welcome
Very clear, and simple explanation.
Glad it was helpful!
great video, I understand what idempotency operations entails, thank you
Well explained. Thank you!!
You're very welcome Santosh! Glad you enjoyed.
this video entails very good explanation and this also entails complex understanding.
Thank you!
Awesome video. Very clearly explained.
Thanks!
Thank you so much for the post.
Good work.
Keep it up.
You're very welcome Raju!
Prepping for Amazon TPM interview and this is so helpful!
Thanks Tamara and good luck on your interview! Make sure you focus on those leadership principles !
Great explanation!
Really good work man... such a detailed video...
Thanks Sofia! Glad you enjoyed :)
Great explanation..! Thanks 🙏🏼
You're very welcome Dor!
Very well explained. Thank you
You're very welcome!
Vids are awesome, really enjoy them. Interesting that you didn't touch on the lack of thought to database design, indexing and maintenance etc as a way to improve performance. Interested to know why? Especially given the cost of scaling in serverless environments.
Great job on this one, I came here to know more about sharding, but I learned lots of useful information before you even dived into the topic ;)
Glad it was helpful!
clear and concise. subscribed
Thanks Libert and welcome!
Great explanation. Thank you
You are welcome!
superb explanation of DB scaling & sharding & W/R databases for a non DB person ;)
Really useful content! Keep it up!
Thanks so much Simone!
Great vidoe, keep up the good work
Are there any database tools that make this easier? Couldn't someone write some software to create a wrapper around a sharded DBMS that could handle the routing and re-sharding with a given hashing key?
Thanks a lof for this video!
Watched some of your random videos on sys design, and now im hooked. Great content!
Thanks so much J! Glad you enjoyed!
@@BeABetterDev yes
très bien expliqué, merci
Very well formed content .. thanks 🙏
You're very welcome!
Superb explanation 😍
Great content, thank you
Your videos are awesome! Thanks
Thanks ray!
good explanation!!
Well explained!!
Thank you!
Nice tutorial. Wonder in real word scenairo, is the routing layer something sits in the application code or it's implemented on the database side?
Thank you, Dude!!!
Awesome! Thanks a lot!
You're very welcome
Love longer videos ❤
great explanation thank u so much
You are welcome!
Thank you that really helpful great video
You're very welcome Tran!
Thank you so much
Great vedio please make vedio on opsmanager installation on production environment
Great video. Thank you. I just have a question about routing for the determining the shards. Is it always necessary? I was thinking that you could just do modulus on the id to get the shard number instead (eg: customer_id: 12, num_of_shards = 4 so the shard would be 12 % 4 = 0). That way you don't have a single point of failure on the router. What are the downsides to this approach vs router end-point ?
Hi there this is a great point thanks for sharing. The problem with using modulus is that it can get difficult to change the assignment of data to shard if you need to re-shuffle your data. With a single table acting as the authority, this can be done trivially.
God bless you, sir ✌️
very good explanation, thank you
one point is not clear - do we really have advantage of availability / fault tolerance, in case we have an intermediate layer that routes the requests? for me it is like the same, isn't it?
excellent video
Great video! But how do we handle foreign keys in sharding?
Much thank you for your great TH-cam help. I am new to Excel and Chatbot. How can I migrate the Excel database, export it from Microsoft Azure WebApp, and import it into AWS Chabot? Keep having errors missing QID and others on the AWS Chabot console. Please help show me the fastest way to convert the Excel and make it compatible with AWS Chatbot?
thank you!
Great content!
Thanks so much Shivakumar!
Great content
I know you have had other dynamodb videos here but would it be possible to have a more in depth video dealing with sharding in dynamodb and also utilizing this with python/boto3 vs the cli? I know it's not really the same type of sharding per se but this video reminded me that I am interested in seeing that kind of thing
Hey HeavensMeat! You're suggestion is a great idea for a new video idea, thanks you! I'll work on incorporating this into my todo list. Cheers!
Which is better architecture, microservice or using single database n use sharding later when it scales?
Excellent !
Thank you so much Vikas!
OMG, I just Sharded myself!
How come I didn't find your channel before?
Thanks for the video
You're very welcome Fethi!
Thanks
amazing video!! Understood almost everything and am not a it guy.. the only thing I did not get is the difference between partition mapping and routing :(
@BeABetterDev What if I were to opt for synchronous replication for my read replicas? Wouldn't that provide me with a high level of consistency (strong consistency) between the master node and the replica nodes? Besides, AWS RDS provides async replication for read replicas, does that mean it is eventual consistent? If so, if I am building an application that needs to opt in strong consistency, shouldn't I use AWS RDS read replicas then? What would be an alternative option to that?
Great vid! I have a question. In massive distributed systems (more read intensive than writes) where hits to your database are really expensive and they are using some form of a caching layer which stores the most frequently accessed data - does the problem of routing go away? Because in this case any writes to the database would mean that you’re invalidating the cache, and reads are done from the caching layer, so even though you may have horizontally partitioned dbs below, they don’t really have to worry about how to route the incoming request for data? I hope my query makes sense.
Same question dude
Even if you have a caching layer serving most of your reads, your cache never stores everything in your DB. For this reason you will still need to solve for reading from the db whenever you have "cache misses," which means the need to retrieve the data from the correct shard still exists (and will require mapping/routing).
very well explained
Thanks so much!
A more interesting concept though is how you generate these unique id's that are used in the sharding / partitioning and ensure uniqueness
I love this breakdown, but it does somewhat leave me wondering when Sharding would be a good vs a bad idea. The cons seem pretty hefting in comparison to the pros.
It would have been nice to run through a few specific different use cases and when one strategy would be better than another.
What I always miss in these videos is, doesn’t introducing a routing layer just kick the can down the road? Now you have all traffic going to a singular routing node, which is not scalable and can fail. What happens when you need to scale the routing node?
Can we scale up and scale down the storage of database as per daily requirement using sharding?