Thanks for your channel Naren! Brings back my love for computer science. We need more such teachers that can break things down and explain it as simply as you have done here.
Great system design. I really wish he explained why file change sets need to be ordered and consistent, in which led him to use a relational database for the metadata. If you look at his design for google docs, it doesn't even use a relational database for massively concurrently updated files.
The best part I like about your videos is you do a lot of research to put the information from various sources about a topic into one place. You are our Edgestore ;)
Another reason to use async queues: one cannot assume that only a single file will be uploaded. There could be a case in which multiple files could be uploaded and a queue ensures that chunks do not get mixed with each other. I guess one can also talk about failover (what happens when a chunk gets lost during transmission/gets corrupted) but that might not be required. Edit: NVM he covers this case as well LOL. Love the depth he goes into when covering different components.
Hello Naren! your channel is a goldmine. I've learned quite a lot. Please consider creating content that dives deep into data models/schemas/datasets. Thanks 🙏
Clients described at 18:26, taking an example of Google Drive, refer to the various "Backup and Sync" desktop clients which you might have active on multiple devices. All these clients keep listening to a messaging queue. In case one device makes changes to a file, the change is propagated to S3 and all clients are notified of this by publishing the change to the messaging queue which they are listening to. The client which is the originator of the change doesn't care but other clients do and when they know of a change they update their local copies (download the whole file if not present). Update: It's not just one Q2. Each client will have its own queue on which the change is broadcasted. This is to have an asynchronous behaviour wherein the client can be offline for a period and then when it is online it starts listening to the queue for any changes This is my understanding. Correct me if I'm wrong
honestly as a swe working at dropbox, i don't feel like this is an answer i am looking for. It misses a lot of important stuff like how do you design your database schema for storing the metadata and how would your sync protocols looks like? what if there are write conflicts during sync how do you deal with that? and the search engine part i guess is the least likely bonus question i'll ask in an interview(probably makes more sense in design twitter) no offense to Narendra, i think you put in a lot of effort/research into this and even referenced dropbox's blog post on network edge infra. but i think this's a problem to almost all of these youtube system design videos, like, yes you will learn a little bit here and there, but it's not the same as a real interview and don't expect to memorize some sys design solution and pass the interview. better ways to learn system design: read DDIA, web scalability for startup engineers, take a distributed system class listen to real mock interviews if you somehow can(or some faang engineer does these mock interviews and post them somewhere i guess) design and implement projects at your job if you have the opportunity
WTF. only 633 likes out of 38,663 views for this gold? Come on viewers, you are beholden for this guy who is putting enormous effort to share knowledge beyond his boundaries.
Needs an explanation of how exactly does one detect which chunk was changed. Because your applications, video editor, for example, doesn't know anything about chunks, it doesn't change a chunk, it changes your file. It's up to your Dropbox client to figure out which chunk the change corresponds to. And that is not immediately obvious especially for huge binary files.
Hey Naren. Great job! Few questions for you. 1. Why can't we expose a single service which takes chunks of data and make metadata entry into database and also stores chunks to S3 instead of client calling both services? 2. From your design, if sync service pushes notifications to a topic are we maintaining dedicated topics/partitions for different clients? Or are we pushing notifications via Websockets/HTTP Polling? Few comments: 1. If clients go offline they can still come back and establish connections via Websockets? 2. We can't have 'n' number of topics because creating Kafka topics/JMS queues need infrastructure support and is a costly operation. Also creating partitions in a live system is a costly affair. Pls let me know if I'm missing anything.
Salaams and respect from Pakistan for you sir! You are a hard working and a smart individual who is helping the IT community across the world using whatever best resources you have. Keep up the good work - Keep posting them system design videos. God Bless!
33:56 Here you threw the problem that we need to rebalance/re-shard as we get more and more data in one shard, but the subsequent mentioned approach "edgestore" does not seem to solve this, does it? It seems like the edge wrapper simply provides a better interface for developers to read/write data. How does the "edgestore" help in regards to the data sharding parts?
While explaining why we need queue instead of http call to sync service you mentioned we need it as client may not always be connected. My question is if client dont connect to internet for example, even that message also cant be transmitted to queue right ?
Without reference to original paper “Designing a Dropbox-like File Storage Service” by Alejandro Ramirez, Fariborz Khanzadeh, Hassaan Bukhari. this is unfair.
I am not quite clear about the response queue. Is it necessary? If each client maps to a response queue, and what if the client never comes back? Are we still posting messages to its queue? Meanwhile, why not just let each client periodically check the diff between the local metadata vs. the latest metadata? By doing this, we can get rid of the response queues, right?
What happens to the chunks when I add data to the file that would be contained in the first chunk causing it to go over it's limit? Wouldn't that cause a cascading effect where every chunk spills over into the next chunk? Our small change in one chunk would cause changes in every chunk no?
@@sudhasravan92 Haha at least at this point it was not just me scratching my head. Jokes apart, seriously I once had a discussion with my coworker why git scm was not being used, for that he reminded me how git works. which is by storing delta/diff between two files, so that when a file is modified, only the delta info is uploaded or downloaded. BUT, he explained me that this is exclusive to TEXT ENCODED files, and not for BINARY files because git can in no way know what is the delta because actual data is binary (such as .exe, .obj, .dat, .class etc). He confirmed that in case of binary files, git actually stores the new file completely. so this is equivalent to storing old file + new file which doubles size of storage required. HENCE git is not intended to store BINARY Files where delta info can't be determined. Considering this theory, you could see that the chunking current file to be uploaded can save you in terms of network errors so that you can re-upload erroneous chunk again, but it is completely not helpful in terms of using as delta information. Because when the file is modified, the whole file can't be chunked again as how the previous version was chunked and compared with previous version of chunks in 1:1 manner, nor it can be variably chunked such that we can deduce the exact chunk that has changed considering file is binary where data could be machine code(exe) of a processor. If someone can point me "THE OBVIOUSNESS" of the chunker design shown here and its purpose/usefulness, I would be much thankful!
Truly amazing. Hats off to you. 🙏😍 Request you to upload more of such videos. It would be too awesome if we can have a system design tutorial for beginners and how to improve.
Man, I wish I discovered your channel sooner. I recently failed on a system design interview, Dropbox system design particularly. Thanks for your work. I will study every single of your video and prepare myself for my next interviews.
Nice explaination Naren. Why is messaging service sending information to clients. It should rather be pull model on client's front, where they periodically pulling from Server the chunks. It should be based on last synched chunk_id. So, even if any client is unavailable for an interim duration he can synch based on last synched chunk_id.
How does this work ? Lets say some client had last synched the chunk id = 10 of a particular file (this file has 10 chunks). Now lets say chunk_id = 5 has changed for this particular file. How will the other clients know that they need to get chunk_id = 5 from the object store ? Yoir idea only works if the file increases in size ,thereby increasing the chunk_id. Please correct me if I am wrong
@@psn999100 you store the last time there was a modification in the client #1, and let’s say another client #2 updates a chunk, then you update the latest modification time in the server. When client #1 asks if there has been any changes using the last time of client #1, the server compares it to its last modification time (done by client #2) and lets the client #1 know that there are some changes that client #1 is missing, these changes are the changes done by client #2.
separate queue for each client doesn't sound good additionally we are using queue as persistence storage which should be avoided because a large number of messages can pile up in queue without any proper ordering. instead, the client side can call the sync service to fetch the latest files index for the user
I have a question about the response queues. At the beginning you mention dropbox has 500 million users. If we assume a user has 3 devices on average, that leaves us maintaining 1.5 billion response queues. Am I misunderstanding something?
Firstly, thanks for the video. it would have been interesting to know how the Edge Wrapper achieves transaction isolation level without explicit locking/transaction.
just fyi, cassandra consistency model provides a higher chance of reads being consistent, but doesn't provide true linearizability. This is why it's better to use terms like linearizability and not consistency as DB providers can play games with their definition of "consistency". Cassandra and similar nosql variants are basically partitioned key value stores in disguise and cannot ever compete with a true relational database. Also, even within relational databases, configuring isolation levels is pretty important, and it's easy to get tripped up there.
Those are great videos that u r doing. Can you please start a course about system design basics n how to build from scratch to advanced level. Please do that course I would love to buy. Thank you 😀
11:25 You can't just break a file like that, what if one of the parts' size increased 100 times? what if it's a binary file and you just synced something that broke it?
all chunks are going to be the same size. If the file gets bigger you get more chunks. now if its not a text file and you can not do good diffs, the client needs to split the whole file with the same algorithm again into chunks. Then the client compares the "new meta data" with the "old meta data" and just sends the changed junks to the server
@@sniGGandBaShoR so basically what you are saying is, after the first 2 kb of a 20GB binary file there is 10kb data added, then all the 40 chunks of 500mb are now obsolete as the first chunk is already modified and rest of the chunks are displaced by 10kb so wouldn't match with previous version either. Eventually you are simply uploading full 20GB :( I am eager to know how you propose to solve the "chunker problem". What I can think of are: A binary diff tool which can find the diff in bytes from previous version to current version such as a patch. if such a tool is available, still you need to consider the possibility that original file cannot be modified in any means. So the uploader software which has this "chunker" component needs to basically create chunks of data physically on the disk? ie (use another 20GB?) or even if it does process chunk by chunk. Does it going to build an in-memory file chunk marker meta structure(like seek pointer data without actually chunking the file) such that chunk data of each file is processed in the client system? How is it going to respond to 10 following 20 GB files waiting to be uploaded while it computes this chunk data itself in like next 20 mins? then upload it in next 2 mins?
@@sniGGandBaShoR At least I have seen gdrive and dropbox and I don't think they do this chunk transfer for "obvious reasons" coz when I say upload a 2 GB video then do a small edit to cut out last 1 min and reupload the new 1.9GB video, it still takes full time that was required for initial 2GB.
15:38 hahahahaha, I do that sometimes too haha. I pause then forget to clip the video or I just can’t clip the video since noise is in a critical segment.
Thanks, you are doing a great job. Also, It would be really helpful if you could run the whole flow once at the end. So that we don't have to watch the full video when revisiting the video for the second time.
Great Video Narendra. I have a question. Don't you think there won't be any compression and encryption of data happens? I think this help in both Bandwidth and security.
Thanks Narendra for this great resource of designing DropBox, could you please help me with following few doubts: 1. How we can implement this application to resolve any conflicts. For example, I made few changes in a document and same is synced into server and but other clients were offline for quite some time. Then, I started using another client which is offline; however, I made significant changes into it and then only realised that I am offline. Then what would happen, I don't want to loose any of my changes and the mentioned document is not a simple text file. 2. Should chunk size be dynamically determined depending upon an individual file size or it would be fixed for the entire application. If it is fixed, and then we have made necessary changes in a document's intermittent chunk number (say suppose 2nd chunk among 10 chunks in total). Now chunk size of 2nd chunk got tripled, should we break it and reorder all the chunks (my point is if there are 1000 chunks, then re-ordering would be an issues) or keep it as it is, what are your comments on this? Thanks for your inputs in advance.
Great work Narendra. I'm learning a lot from your videos. I have gone through almost all your system design videos. Just checking if you can create one on a Saas product like Salesforce. I didn't find any good video on Salesforce / Shopify like services.
Hello Narendra, Could you please make a video to design "google photos" like app? Or what architectural changes you would do in this existing design of drop box to limit it to "google photos"? By the way, your video has been real source of knowledge!
Hey I really like the explanation and concept of solution you provide. Can you make a video of TH-cam system design. As there is no video on TH-cam yet.
Chunks idea is good, but your statement that if we change only some bytes we will only have to upload that chunk has a problem. If you add X characters to 4th chunk, all the following chunks change as well, they all will be shifted by X characters. So when your script checks the hash sum of a chunk, it will have to upload [4;n] chunks. Unless we can adjust how we split a file into chunks.
I think you just need to know how HDFS systems chunk the data into its default size. It won't separate the data from Line -L and Column - C 1L-1C to 10L 12C -chunk1 10L-13tC to 25L-30C - chunk2 . . so........
Great job Nagendra, look forward to seeing more interesting content from you. A part of system design it would also be nice if you could do a couple of class design and DB design examples. Design a chess game (all the classes and design patterns) or Design the database schema for instagram would be good examples.
I really like the idea of breaking the file into chunks but why not use document diff libraries to be sure to only capture the changes that have taken place?
calculating diff is a complicated in a collbartive environment. He was explaining in one video about operational transformer. we can use libraries but how that works also we have to know.
I'm not convinced by the argument for using queues in the design. If clients need to obtain the latest changes, they will need to establish a connection of some sort to some service. You could make a REST call asking for the latest data which could do a search against the metadata DB, or you could call a service that reads from the queue. I think the main argument for using a queue is latency. Your read/write path might take a long time, so could stand to gain by doing append only writes to a queue and having latest metadata responses pre-populated in multiple queues.
Great video!! Wanted to cross check is S3 block storage? I thought S3 is object storage and EBS is block storage. I didn't know what the 2 were but luckily stumbled upon 19:00 and started googling. Thanks!
@26:14 : If the devices are offline, for that I think, we are using Clients in our services...I think queues will be used for scaling. @33:19 : Seems statement got incomplete during editing?
Hey Narendra, Great video. I am learning a lot. In the beginning of the video you talked about designing the system for 10 million users. Is this coming in another video? How is the sizing for required resources done. I am curious. Thanks mate.
Trying to understand what is reading the Database? It seems the sync service is the only one interacting with the database and then pushing the same message it received back to the response queue?
Narendra L Design looks good. But i doubt current design will work using browser as clients. Could you please update the documentation on how to optimize downloads upload/download using browsers as google chrome etc. Is it possible to add client side logic split and merge chunks in google chrome browser ?
Naren, how does the message queue send http protocol to other clients is it polling? I think it should be a different protocol. I think it should be a websocket right?
Would be helpful if you also include the sql schema for important tables such as entity table, chunk table etc. At 31:43 you only show chunk table json but since this is mysql, not fully clear. How would entity (file/folder) table look like?
idea scope 1:38
scale 2:10
HLD 2:41
problem to solve 4:55 6:57
solution 10:41
metadata file 15:26
HLD 17:38
messaging service detail 25:01 device sync feature
metadata handling 28:40
metadata schema 31:48
edge store usage to serve metadata 36:16
search feature 40:01
Thanks for your channel Naren! Brings back my love for computer science. We need more such teachers that can break things down and explain it as simply as you have done here.
Give this man the credit he deserves 👏🏼👏🏼👏🏼
Great system design. I really wish he explained why file change sets need to be ordered and consistent, in which led him to use a relational database for the metadata.
If you look at his design for google docs, it doesn't even use a relational database for massively concurrently updated files.
Yes. He explained google docs using operational transformation.
Great work Narendra. This is the best video I have found so far on TH-cam on the DropBox architecture.
Enjoyed this video more than others because of the cute doggo interruptions. :) Thank you!
The best part I like about your videos is you do a lot of research to put the information from various sources about a topic into one place. You are our Edgestore ;)
Another reason to use async queues: one cannot assume that only a single file will be uploaded. There could be a case in which multiple files could be uploaded and a queue ensures that chunks do not get mixed with each other. I guess one can also talk about failover (what happens when a chunk gets lost during transmission/gets corrupted) but that might not be required.
Edit: NVM he covers this case as well LOL. Love the depth he goes into when covering different components.
15:39 LMFAO! great video man, you are my go to for system design prep
Hello Naren! your channel is a goldmine. I've learned quite a lot. Please consider creating content that dives deep into data models/schemas/datasets. Thanks 🙏
Sure, Thanks
@@TechDummiesNarendraL Do you have code for this explained system?
Your explanations and approaches in explaining these System Design Problems is absolutely phenomenal.
Clients described at 18:26, taking an example of Google Drive, refer to the various "Backup and Sync" desktop clients which you might have active on multiple devices. All these clients keep listening to a messaging queue. In case one device makes changes to a file, the change is propagated to S3 and all clients are notified of this by publishing the change to the messaging queue which they are listening to. The client which is the originator of the change doesn't care but other clients do and when they know of a change they update their local copies (download the whole file if not present).
Update:
It's not just one Q2. Each client will have its own queue on which the change is broadcasted. This is to have an asynchronous behaviour wherein the client can be offline for a period and then when it is online it starts listening to the queue for any changes
This is my understanding. Correct me if I'm wrong
honestly as a swe working at dropbox, i don't feel like this is an answer i am looking for. It misses a lot of important stuff like how do you design your database schema for storing the metadata and how would your sync protocols looks like? what if there are write conflicts during sync how do you deal with that? and the search engine part i guess is the least likely bonus question i'll ask in an interview(probably makes more sense in design twitter)
no offense to Narendra, i think you put in a lot of effort/research into this and even referenced dropbox's blog post on network edge infra.
but i think this's a problem to almost all of these youtube system design videos, like, yes you will learn a little bit here and there, but it's not the same as a real interview and don't expect to memorize some sys design solution and pass the interview.
better ways to learn system design:
read DDIA, web scalability for startup engineers, take a distributed system class
listen to real mock interviews if you somehow can(or some faang engineer does these mock interviews and post them somewhere i guess)
design and implement projects at your job if you have the opportunity
It depends on the system design interview you attend.
This should be pinned
You don't have studio but you are delivering better content than those who have studio.
WTF. only 633 likes out of 38,663 views for this gold? Come on viewers, you are beholden for this guy who is putting enormous effort to share knowledge beyond his boundaries.
Awesome video that comes down to details for real design not just for interviews 😄
Give this man Bharat Anmol Ratna : ]. Thanks for SD series it helps us broaden our thinking and not just defect fixing and small CR.
Needs an explanation of how exactly does one detect which chunk was changed. Because your applications, video editor, for example, doesn't know anything about chunks, it doesn't change a chunk, it changes your file. It's up to your Dropbox client to figure out which chunk the change corresponds to. And that is not immediately obvious especially for huge binary files.
Hash computation can help
@@jyotsnamadhavi6203 Yes, correlation ID is used in Kafka.
Hey Naren. Great job! Few questions for you.
1. Why can't we expose a single service which takes chunks of data and make metadata entry into database and also stores chunks to S3 instead of client calling both services?
2. From your design, if sync service pushes notifications to a topic are we maintaining dedicated topics/partitions for different clients? Or are we pushing notifications via Websockets/HTTP Polling?
Few comments:
1. If clients go offline they can still come back and establish connections via Websockets?
2. We can't have 'n' number of topics because creating Kafka topics/JMS queues need infrastructure support and is a costly operation. Also creating partitions in a live system is a costly affair. Pls let me know if I'm missing anything.
Though this video is a good starter, its gets wrong at multiple places
A Great Video on Understanding file storage service design like dropbox, Preparing for an interview and this content is helpfull
Shit, it's fucking perfect explanation. Thanks for all these stuff.
Amazing no nonsense serious designs which are really good hatsoff bro 👍 keep doing good work
Salaams and respect from Pakistan for you sir! You are a hard working and a smart individual who is helping the IT community across the world using whatever best resources you have. Keep up the good work - Keep posting them system design videos. God Bless!
33:56 Here you threw the problem that we need to rebalance/re-shard as we get more and more data in one shard, but the subsequent mentioned approach "edgestore" does not seem to solve this, does it? It seems like the edge wrapper simply provides a better interface for developers to read/write data. How does the "edgestore" help in regards to the data sharding parts?
Yeah Naren, would like to know the answer to this - how is the cost of denormalization required due to sharding reduced by edgestore?
Great system design and clear explanation, thank you !
Excellent the way of explaining the concept.
and really enjoyed the the dogs pictures while barking in the mid of presentation. 🙂👍
While explaining why we need queue instead of http call to sync service you mentioned we need it as client may not always be connected. My question is if client dont connect to internet for example, even that message also cant be transmitted to queue right ?
Very informative. You have covered each layer like front end, Middle tier and database layer effectively. Thanks
Really you done a good & great job annaiah.....Awsome explanation,tq☺️
Sirji excellent video
Very nice video! Please do an Instagram system design for the next one! Thank you!
Without reference to original paper “Designing a Dropbox-like File Storage Service” by Alejandro Ramirez, Fariborz Khanzadeh, Hassaan Bukhari. this is unfair.
Awesome! Loved the explanation and learned a lot. Last part of the search design for this service could be expanded into another video.
Shooting a video for serchengine design
I am not quite clear about the response queue. Is it necessary? If each client maps to a response queue, and what if the client never comes back? Are we still posting messages to its queue? Meanwhile, why not just let each client periodically check the diff between the local metadata vs. the latest metadata? By doing this, we can get rid of the response queues, right?
Is the logic same for JPG files (images or videos)? Is it different to stitch together a text file vs video file?
oh, its in bytes is it. chunks are in 1/0 format. ok got it. facepalm :)
videos should be encoded and broken into chunks
What happens to the chunks when I add data to the file that would be contained in the first chunk causing it to go over it's limit? Wouldn't that cause a cascading effect where every chunk spills over into the next chunk? Our small change in one chunk would cause changes in every chunk no?
Exactly!! I have been struggling with the same question for the last few days but could not find an answer anywhere!
@@sudhasravan92 Haha at least at this point it was not just me scratching my head. Jokes apart, seriously I once had a discussion with my coworker why git scm was not being used, for that he reminded me how git works.
which is by storing delta/diff between two files, so that when a file is modified, only the delta info is uploaded or downloaded.
BUT, he explained me that this is exclusive to TEXT ENCODED files, and not for BINARY files because git can in no way know what is the delta because actual data is binary (such as .exe, .obj, .dat, .class etc).
He confirmed that in case of binary files, git actually stores the new file completely. so this is equivalent to storing old file + new file which doubles size of storage required.
HENCE git is not intended to store BINARY Files where delta info can't be determined.
Considering this theory, you could see that the chunking current file to be uploaded can save you in terms of network errors so that you can re-upload erroneous chunk again, but it is completely not helpful in terms of using as delta information.
Because when the file is modified, the whole file can't be chunked again as how the previous version was chunked and compared with previous version of chunks in 1:1 manner,
nor it can be variably chunked such that we can deduce the exact chunk that has changed considering file is binary where data could be machine code(exe) of a processor.
If someone can point me "THE OBVIOUSNESS" of the chunker design shown here and its purpose/usefulness, I would be much thankful!
The content is always great from this channel, but if you can use a microphone while talking that will bring the video to the next level.
Truly amazing. Hats off to you. 🙏😍 Request you to upload more of such videos. It would be too awesome if we can have a system design tutorial for beginners and how to improve.
Sure, Some time soon
Man, I wish I discovered your channel sooner. I recently failed on a system design interview, Dropbox system design particularly. Thanks for your work. I will study every single of your video and prepare myself for my next interviews.
which company ?
@@lakshminarayanansairam2739 Ledger. The company that builds crypto wallets.
So clear and easy to understand, keep going!
Nice explanation. Insightful
Nice explaination Naren. Why is messaging service sending information to clients. It should rather be pull model on client's front, where they periodically pulling from Server the chunks. It should be based on last synched chunk_id. So, even if any client is unavailable for an interim duration he can synch based on last synched chunk_id.
I agree
How does this work ? Lets say some client had last synched the chunk id = 10 of a particular file (this file has 10 chunks). Now lets say chunk_id = 5 has changed for this particular file. How will the other clients know that they need to get chunk_id = 5 from the object store ? Yoir idea only works if the file increases in size ,thereby increasing the chunk_id. Please correct me if I am wrong
@@psn999100 you store the last time there was a modification in the client #1, and let’s say another client #2 updates a chunk, then you update the latest modification time in the server. When client #1 asks if there has been any changes using the last time of client #1, the server compares it to its last modification time (done by client #2) and lets the client #1 know that there are some changes that client #1 is missing, these changes are the changes done by client #2.
We may need to pull only when client connects then for further updates queue can be used that can help us avoid constant polling to server
Wow. Amazing. U r doing a grt job.
Wonderful Explanation...!! Thanks for the work Naren.
separate queue for each client doesn't sound good additionally we are using queue as persistence storage which should be avoided because a large number of messages can pile up in queue without any proper ordering. instead, the client side can call the sync service to fetch the latest files index for the user
Amazing .. I am new in system design and I've learned a lot.. Thankyou so much
I have a question about the response queues. At the beginning you mention dropbox has 500 million users. If we assume a user has 3 devices on average, that leaves us maintaining 1.5 billion response queues. Am I misunderstanding something?
I think this solution is not scalable.
thank you for the video
it gets the very general idea about how it works
but without important details though
once again thanks
Love your videos...Thank you so much for sharing
THE BEST OF THE BEST -> PLEASE, CONTINUE YOUR CHANNEL!
Thank you so much. The best system design video on this topic.
Firstly, thanks for the video. it would have been interesting to know how the Edge Wrapper achieves transaction isolation level without explicit locking/transaction.
What if we send the diff only, what git does. Storing a tree like structure of changes.
just fyi, cassandra consistency model provides a higher chance of reads being consistent, but doesn't provide true linearizability. This is why it's better to use terms like linearizability and not consistency as DB providers can play games with their definition of "consistency". Cassandra and similar nosql variants are basically partitioned key value stores in disguise and cannot ever compete with a true relational database. Also, even within relational databases, configuring isolation levels is pretty important, and it's easy to get tripped up there.
Those are great videos that u r doing. Can you please start a course about system design basics n how to build from scratch to advanced level. Please do that course I would love to buy. Thank you 😀
excellent video , thanks a ton .
pls make a video on system design for decentralized applocations on ethereum and ipfs (like decentralized uber)
You are the best! Thank you so much for explaining this so nicely!!!
Doing great job Naren, keep up the spirit 👍🏻
Very good content. I loved the dog barking.
11:25 You can't just break a file like that, what if one of the parts' size increased 100 times? what if it's a binary file and you just synced something that broke it?
all chunks are going to be the same size. If the file gets bigger you get more chunks.
now if its not a text file and you can not do good diffs, the client needs to split the whole file with the same algorithm again into chunks. Then the client compares the "new meta data" with the "old meta data" and just sends the changed junks to the server
@@sniGGandBaShoR so basically what you are saying is, after the first 2 kb of a 20GB binary file there is 10kb data added, then all the 40 chunks of 500mb are now obsolete as the first chunk is already modified and rest of the chunks are displaced by 10kb so wouldn't match with previous version either.
Eventually you are simply uploading full 20GB :(
I am eager to know how you propose to solve the "chunker problem".
What I can think of are:
A binary diff tool which can find the diff in bytes from previous version to current version such as a patch.
if such a tool is available, still you need to consider the possibility that original file cannot be modified in any means.
So the uploader software which has this "chunker" component needs to basically create chunks of data physically on the disk? ie (use another 20GB?) or even if it does process chunk by chunk.
Does it going to build an in-memory file chunk marker meta structure(like seek pointer data without actually chunking the file) such that chunk data of each file is processed in the client system?
How is it going to respond to 10 following 20 GB files waiting to be uploaded while it computes this chunk data itself in like next 20 mins? then upload it in next 2 mins?
@@sniGGandBaShoR At least I have seen gdrive and dropbox and I don't think they do this chunk transfer for "obvious reasons" coz when I say upload a 2 GB video then do a small edit to cut out last 1 min and reupload the new 1.9GB video, it still takes full time that was required for initial 2GB.
for the length of video, i learned a ton
believe me, ur channel will gonna have 50K+ subscribers within 3 months, keep up the good work
:) I wish, Thanks
Fabulous videos, excellent information and lots to learn ! Dogs were hilarious.
15:38 hahahahaha, I do that sometimes too haha. I pause then forget to clip the video or I just can’t clip the video since noise is in a critical segment.
Hey back to back SWE. :D
@@karthikmucheli7930 hey man, wassup
@@bephrem yo, you are great. Like all your videos. :)
Thanks, you are doing a great job. Also, It would be really helpful if you could run the whole flow once at the end. So that we don't have to watch the full video when revisiting the video for the second time.
Do you mean there will be a queue maintained for each client? Actually two, one request and one response queue. Is this design efficient?
Amy Zeng I thought the same thing. Waiting on a comment from him about it
Thank you so much. This was fascinating!
you look really good and confident
Just curious about the chunks, are you assuming the client to have the knowledge of and be able to handle various file formats?
Amazing video, Very detailed and to the point!! If possible, please add Fault tolerance and Security related usecases to be incorporated in the design
Great Video Narendra. I have a question. Don't you think there won't be any compression and encryption of data happens? I think this help in both Bandwidth and security.
Thanks Narendra for this great resource of designing DropBox, could you please help me with following few doubts:
1. How we can implement this application to resolve any conflicts. For example, I made few changes in a document and same is synced into server and but other clients were offline for quite some time. Then, I started using another client which is offline; however, I made significant changes into it and then only realised that I am offline. Then what would happen, I don't want to loose any of my changes and the mentioned document is not a simple text file.
2. Should chunk size be dynamically determined depending upon an individual file size or it would be fixed for the entire application. If it is fixed, and then we have made necessary changes in a document's intermittent chunk number (say suppose 2nd chunk among 10 chunks in total). Now chunk size of 2nd chunk got tripled, should we break it and reorder all the chunks (my point is if there are 1000 chunks, then re-ordering would be an issues) or keep it as it is, what are your comments on this?
Thanks for your inputs in advance.
There is a version control system called git which can help to accomplish the doubts you have.
Great work Narendra. I'm learning a lot from your videos. I have gone through almost all your system design videos. Just checking if you can create one on a Saas product like Salesforce. I didn't find any good video on Salesforce / Shopify like services.
Hello Narendra,
Could you please make a video to design "google photos" like app? Or what architectural changes you would do in this existing design of drop box to limit it to "google photos"? By the way, your video has been real source of knowledge!
Great job, Naren! Love your work. Keep it up!
Can you create videos on data models/schemas/datasets as has been asked some comment below as well.
Hey I really like the explanation and concept of solution you provide. Can you make a video of TH-cam system design. As there is no video on TH-cam yet.
Chunks idea is good, but your statement that if we change only some bytes we will only have to upload that chunk has a problem. If you add X characters to 4th chunk, all the following chunks change as well, they all will be shifted by X characters. So when your script checks the hash sum of a chunk, it will have to upload [4;n] chunks. Unless we can adjust how we split a file into chunks.
I think you just need to know how HDFS systems chunk the data into its default size. It won't separate the data from
Line -L and Column - C
1L-1C to 10L 12C -chunk1
10L-13tC to 25L-30C - chunk2
.
.
so........
Great job Nagendra, look forward to seeing more interesting content from you. A part of system design it would also be nice if you could do a couple of class design and DB design examples. Design a chess game (all the classes and design patterns) or Design the database schema for instagram would be good examples.
Sure I will as soon as I get more time to work on videos.
sweetest part of the video at 15th Min
Great system design video. Thank you !!!!
I really like the idea of breaking the file into chunks but why not use document diff libraries to be sure to only capture the changes that have taken place?
calculating diff is a complicated in a collbartive environment. He was explaining in one video about operational transformer. we can use libraries but how that works also we have to know.
I'm not convinced by the argument for using queues in the design. If clients need to obtain the latest changes, they will need to establish a connection of some sort to some service. You could make a REST call asking for the latest data which could do a search against the metadata DB, or you could call a service that reads from the queue. I think the main argument for using a queue is latency. Your read/write path might take a long time, so could stand to gain by doing append only writes to a queue and having latest metadata responses pre-populated in multiple queues.
First it was awesome explanation, one question clicked in my mind how git ver works with file history and lines changes detected?
The most handsome tech guy I have found in youtube! Thanks a lot !
Great video!! Wanted to cross check is S3 block storage? I thought S3 is object storage and EBS is block storage.
I didn't know what the 2 were but luckily stumbled upon 19:00 and started googling. Thanks!
@26:14 : If the devices are offline, for that I think, we are using Clients in our services...I think queues will be used for scaling.
@33:19 : Seems statement got incomplete during editing?
@34:00 - @36:00 how ORM wrapper solves the issue of re-sharding or addition of more DBs?
Are you saying there is a response queue for every client?
This is amazing video with lots of details. If you could add more details on which part runs where , it will be complete .
Hey Narendra,
Great video. I am learning a lot. In the beginning of the video you talked about designing the system for 10 million users. Is this coming in another video? How is the sizing for required resources done. I am curious. Thanks mate.
Truly one of the best system design videos on TH-cam. Well done!
Only if you are new to this
Trying to understand what is reading the Database? It seems the sync service is the only one interacting with the database and then pushing the same message it received back to the response queue?
Narendra L Design looks good. But i doubt current design will work using browser as clients. Could you please update the documentation on how to optimize downloads upload/download using browsers as google chrome etc. Is it possible to add client side logic split and merge chunks in google chrome browser ?
The videos are amazing, Very helpful. I have seen all your videos. Thank you so much. Can you please make a video on Designing Amazon Lockers?
Sure, Its in my TODO List
Naren, how does the message queue send http protocol to other clients is it polling? I think it should be a different protocol. I think it should be a websocket right?
You deserve a million subs. please make a system design on Inshorts and Instagram.
Hi Narendra,
What is that first client mentioned which does chunking, indexing and pushing messages to other clients?
Where this client resides?
Great work! I think we should have a block on the client side to reconstruct the document!
Would be helpful if you also include the sql schema for important tables such as entity table, chunk table etc. At 31:43 you only show chunk table json but since this is mysql, not fully clear. How would entity (file/folder) table look like?
Wonderful videos! Learned a lot from your videos.