How indexes work in Distributed Databases, their trade-offs, and challenges
ฝัง
- เผยแพร่เมื่อ 19 ต.ค. 2024
- System Design for SDE-2 and above: arpitbhayani.m...
System Design for Beginners: arpitbhayani.m...
Redis Internals: arpitbhayani.m...
Build Your Own Redis / DNS / BitTorrent / SQLite - with CodeCrafters.
Sign up and get 40% off - app.codecrafte...
In the video, I explained how indexing works in a distributed database like DynamoDB and the challenges it brings. I discussed creating different types of indexes, such as global secondary indexes, for efficient querying. By using global secondary indexes, queries can be optimized by directly accessing the required data shards. I also touched on local secondary indexes for specific query patterns. Maintaining indexes can be costly, and the choice between global and local secondary indexes depends on the query requirements and consistency needs. The video ended with a suggestion to explore further on index creation using B+ Trees.
Recommended videos and playlists
If you liked this video, you will find the following videos and playlists helpful
System Design: • PostgreSQL connection ...
Designing Microservices: • Advantages of adopting...
Database Engineering: • How nested loop, hash,...
Concurrency In-depth: • How to write efficient...
Research paper dissections: • The Google File System...
Outage Dissections: • Dissecting GitHub Outa...
Hash Table Internals: • Internal Structure of ...
Bittorrent Internals: • Introduction to BitTor...
Things you will find amusing
Knowledge Base: arpitbhayani.m...
Bookshelf: arpitbhayani.m...
Papershelf: arpitbhayani.m...
Other socials
I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff.
LinkedIn: / arpitbhayani
Twitter: / arpit_bhayani
Weekly Newsletter: arpit.substack...
Thank you for watching and supporting! it means a ton.
I am on a mission to bring out the best engineering stories from around the world and make you all fall in
love with engineering. If you resonate with this then follow along, I always keep it no-fluff.
Making sure in depth and quality content reaches everyone is what separates you from the rest, who are making money just by dwelling on the surface. The word "thanks" alone can't show the gratitude of many, including me, who gets benefited a lot. I hope this continues forever!!
Hi Arpit
Big FAN!!
From your System design playlist where you explained about the database that was exactly asked me during the interview @ INDIA MART for Technical Lead Position seems like the interviewer and i studied from the same place😅
and from the microservices playlist many challenges were thrown in the Director of Engineering Round i was able to clear both the rounds because of your videos
Thanks a ton !!!
this is such great news 🔥 Many many congratulations Shubham 🙌
Instead of reading multiple blogs, going through your vlog saves time and brings interest back to the topic.
Thanks Arpit, for making such videos. After watching couple of videos on internals of a database, including yours, I have started enjoying learning about how things work in the background. Thanks for your effort!
Very well explained Arpit. Before watching this video, I was literally confused as to why Partion Key is always required to create a LSI. I understood the complete picture of it now 👌
Amazing. Nice detailed explanation!
How creating GSIs will solve the 2 major problems 1. Shard is slow 2. Shard is dead
Thanks for this video, we too are planning to work on Sharding/Partition in few months and would love to take this project.
Which app do you use to write in an present in this (and other) videos? (in iPad)
Thanks for the great content btw!!
Very well explained❤Learned something new today :)
Gonna dig in a little deeper in this.
Absolutely awesome content!
Hi Arpit, thanks for explaining the concept. I have one question regarding global secondary index, what if the secondary attribute cardinality is very high like `created_at` kind of field? Will this sharding the index based on the value scale? I also need to read more about it, but would like to know your opinion.
Sharding can be done in relational or non-relational databases but I think non-relational db are more preffered as they have less overhead for example performing a join operation on sharded db in kind of a nightmare, but since nosql is imperative you specify the join logic in the application code itself and handle it there.
In a nutshell SQL has feature for join but it is hard to make sense when db is sharded , but nql has no such concept so sharding make sense there much ..
what is meant by 'imperative' here? do NoSQL handle joins very differently?
Thanks
Thank you so much Rohit 🙌
fantastic video
Thank you so much!
Hey Arpit, Great content!
Is there a video on distributed sequence generator?
Thanks for great video. I have followup question
1. Data is sharded by author_Id of the respective blog. So if I know the author_Id, I can efficiently find out which which shard has the blog info that we are interested in.
2. Post that you created global secondary index on blog tag (say mySql, GoLang etc). You mentioned, along with secondary index you will also store blogId. Once you get the shards corresponding to blog tag, you will get all the respective blogIds.
Now my question is how will you efficiently find out blogs for given blogId as you partitioned the actual data using authorId? Am i missing something here?
Thanks
The GSI also store author ID and blog ID. Based on this we can fetch all rows
Can we shard data on multiple parameters? I am assuming it would work like, in this case, input a user id and categiory(optional) and output the nodes in which data can be found. I don't sharding the data in this way is correct because this might introduce latency in data retrival. Also if you could create a video about types of sharding like application level sharding, database proxy sharding etc. That would be great!
is an LSI a separate object that the main data itself? Cant we sort the main data itself by author key and the secondary attribute? Meaning inside of each node, the data would be sorted by athor then the secondary attribute.
What I feel about your TH-cam Channel is, even if someone cannot afford your courses and still watches all the videos( which are FREE, btw! ) , they are more likely to ace the interviews.
yes. and also ace their career.
it is just that I go slightly more practical and in-depth than this in my courses helping people build the right intuition.
In first case where we store blog_id(primary_key) in GIS, we will get the list for blog_ids when we try to get for a particular category. Then how will we get to know that in which db shard this blog id resides ? as the shard is based on author id.
Hi Arpit, same doubt here, Could you please help us with this one.
+1
I think there is a confusion between author_id(which he told asa paritiion key) and blog_id (which we are assuming as a partition key).
+1 In that case, again there will be data collection from all DB shards and combining results as blog_ids are spread across multiple shards ! Don't see any difference from actually submitted query... 🤔
Start kaun sa video se karu system design?????
I'm just curious here, what if in the global secondary index instead of row_id (or primary key), we are able to store the page_no (actual hard storage page no)? This will fasten up reads that include GSIs a bit as it essentially skips the step of querying into the data shard and can directly access data using the page number. I understand that the performance difference might not be huge but in some niche over optimized scenarios this might come handy. The downside I believe would be that index creation will take some more time, but imo that can be written off.
Thanks for good explanation.
But is it OK using sql for saving blog data? Isn't ok nosql.
Which spesific database would you recommend to choose saving blog data?
SQL works like a charm. No need to unnecessarily go for NoSQL solutions unless your data becomes massive.
Good video ❤
With Naive implementation, the DB routing layer was firing queries to both the shard and merging the results (scatter gather). how does GSI change that? even now the data still resides on data node. Routine layer will still have to fire queries to both the nodes. How does it solve anything?
The first approach was querying the main shard with the category key which was not indexed, hence more latency.
The second approach was querying the main shard with the primary key (user id) which is always indexed as it is a primary key, hence less latency.
awesome
But how the global second index solves the problem that one of the shards is slow or dead???
It makes pagination and query efficient. If you store the complete data in GSI the. It removes the need to query the data Shards.