System Design Interview: Design Dropbox or Google Drive w/ a Ex-Meta Staff Engineer
ฝัง
- เผยแพร่เมื่อ 4 มิ.ย. 2024
- 00:00 - Intro
02:03 - The Approach
3:18 - Requirements
8:17 - Core Entities & APIs
15:10 - High Level Design
27:33 - Deep Dives
55:55 - Conclusion
A step-by-step breakdown of the popular FAANG+ system design interview question, Design Dropbox or Google Drive, which is asked at top companies like Meta, Google, Amazon, Microsoft, and more.
This question is most commonly asked at Google and Amazon, but shows up a lot at Meta too as well as across all major tech companies.
Evan, a former Meta Staff Engineer and current co-founder of Hello Interview, walks through the problem from the perspective of an interviewer who has asked it well over 50 times.
Resources:
1. Detailed write up of the problem: www.hellointerview.com/learn/...
2. System Design In a Hurry: www.hellointerview.com/learn/...
3. Excalidraw used in the video: link.excalidraw.com/l/56zGeHi...
4. Vote for the question you want us to do next: www.hellointerview.com/learn/...
Checkout the previous video breakdowns:
Ticketmaster: • System Design Intervie...
Uber: • System Design Intervie...
Connect with me on LinkedIn: / evan-king-40072280
Preparing for your upcoming interviews and want to practice with top FAANG interviewers like Evan? Book a mock interview at www.hellointerview.com.
Good luck with your upcoming interviews!
These are the best System Design videos I have found. Great framework for approaching problems, clear explanations, helpful diagrams. And I really appreciate the notes about how insight’s different seniority levels might approach specific parts
Will be the best system design interview channel for sure. Neat content. Keep up with the quality Man!
With my systems design interview coming up, I was looking forward to your video.
It's great as always.
This is really good, irrespective of whether we are taking interview or not, any person looking at this level of explanation and detail would try to picture software differnetly. Thanks for making such videos, would love to see some more
Love the content and explanation. I watched hundreds of videos on system design from last 4-5 years and also have paid subscription from few. I don't have any doubt that, your channel can become premier system design platform in no time if you keep the content quality high ( just like last 3 videos).
Next video, I'd recommend to talk about messaging platform like WhatsApp or FB messenger. There are so many videos on this topic but didn't find any which explain the details and really help in the interview.
Great video for system design preparation.
Methodology, the way he approached the question was good. 5 steps. Pretty good
It's pure gold! specially the parts where you are stopping the debates abouts what db to choose or if the calculations are needed.
The deep dives are the best part.. no one goes that deep and thats actually what matters in an interview
i dont comment much, but for this kind of explanation i gotta give it u. Hands down the best explanation on youtube . pls continue making these kind of videos . This channel will blow up
♥️
Hey Evan , thank you so much for providing the great content. I really live the way you organize and put content across the board. ❤
This channel has the best system design content on youtube. Keep up the good work.
Great content. Thank you for posting. One of the best system design video I have come across for this design.
This is really awesome, thank you. Please keep them coming!
Really good approach. I love how you build up to the full solution. It makes a lot of sense to me and helps me reason these complex systems as well
Thanks for the effort and videos. Easily one of the best in business for mocks and educational material.
Great stuff again!
Not only good for interview but also for daily work
1) Clear and concise structure
2) Weigh trade off rigorously and explain the final decision clearly. Every single component is well though out with real world considerations
one of the best system design preparation video I have seen online.
I just finished reading the article on this lol. I’ll check out the video as well.
👏Bravo, on point as always. Thanks Evan, keep up the good work man!
Hi Evan, Thankyou so much for providing this type of content. plz make a system design video on payment system
great videos, thanks for uploading these. Easily the best content about system design interviews I've found.
I would probably suggest to use merkle trees for the sync functionality, seems like a natual way to diff and sync large file systems
I really like the 5-step structure, it's the best I've seen and it effectively helps me think through the designs in a methodical way.
Right on! So glad it’s useful
this is pure gold thank you so much
the content is good, keep up the good work. expecting more SD videos in more details
Thank you for your effort! Excellent content!! Love it!
For the delta sync bit, probably should go a bit deeper into rechunking for an existing file, to perform the delta sync.
Very easy understand keep continue new videos, thanks you so much
One of the best and detailed explanation.
Thank you so much for this content. Please upload more such videos.
New one later today!
One more comment on this: comparing this to the written content on hellointerview, this one seems more round and well-thought (mainly regarding using S3 notif. on chunk upload completion, which wont work). Would be cool to see it reflected there on the platform! Good job
Good feedback! I'll try to get that updated, particularly by adding sync which I just last minute decided to throw into the video.
one can argue that "supporting 50gb upload file size" is a functional requirement (you placed it under non-functional requirement) - just a call out. great video!
Voted on your website for payment system! Banks love these
Great content!
Thanks for the videos. 47:45 I would like to know your opinion on push approach? By push approach I meant when the File service knows there is a change in a chunk, Sync service will let the client know. And, then the client will send a request to sync/download the chunk.
I have a product architecture interview coming up. I was really looking for some good product architecture/design examples and then came across this. This is very helpful because you talk about the client, user experience, malicious users and relate it to the design decisions made. Thank you!
One question, for a product architecture interview - should we go into more details about the APIs like explicitly write out requests, response, failure/success codes or the amount of discussion you did on APis is enough for senior level?
Can you also tell me what topics/ points would you add over the discussion in this video if this was asked in a product architecture design round. Thanks again!
Thanks great content. Probably the best.
Loved this resouce, thank you so much! But I just wanted to point out that in my interview I was asked about sharing files with other users and I feel like this design concentrated more on just syncing files across multiple devices. In the former, I think we can talk a little more about CDN/other approaches which were hand waved here.
Checkout the write up I linked! I go into sharing there.
Love your channel !
Thank you for this video! It is very helpful.
I do have a question though - lets say the same file is modified by 2 devices at the same time. How do we handle sync in that case?
Big fan of this channel and Evan. Please solve whatsapp/messenger type chat system next if you get some time.
Hey Evan, this content is fantastic, thank you!
I have a question regarding your solution to chunking around the 39 minute mark
When we ask S3 to fetch us a pre-signed URL, do we do that for all our chunks as well? Does this happen on initial request to upload the file (metadata)?
The way the File Metadata entity schema is described, it looks like we have a top-level S3Link, but also chunk-level S3 links embedded in the file metadata, so the upload flow is a little unclear to me
Good question, you're right to be a little confused here. So as I alluded to S3 offers and API called multi-part upload. For this, it requires just 1 presigned url, but, multi-part upload re-stitches the chunks back into a single file in s3, so this does not allow us to send over chunk deltas for syncing.
As a result, we have to upload as chunks manually without relying on multi-part upload. So, long answer, but yes, you'd actually need to request a presigned url for each chunk, I should have made that clearer but tbh was not sure in the moment if multi-part upload could be configured to not re-stitch the file, so I omitted :)
Thank you.
Thanks for the great content! Your video really help me understand the "flow" about a good system design interview (which I do feel very important for staff+ engineers to direct the interview!)
a few general questions:
1. Is it beneficial to mention the selection of specific DB (like dynamo), and mention to use Kafka/Spark for microbatching snapshot update etc. after evaluating the QPS / replication etc.? or keep it at abstract level would give equivalent signals for interviewers?
2. what signals / thinking might make you feel it even beyond staff (just out of curiosity, as feeling it has already been very perfect from the staff requirement)?
1. You'll need to show technical excellence somewhere, these could be two good places. Ideally, you go deep in the places you know well and have hands on experience. If this is where that is, then go for it. If you keep it abstract, then the depth needs to come from somewhere else.
2. Hard to say, Staff candidates can usually teach me something, which is a key sign. They know some part of a system better from work experience then I do, so we can go deep there and I end up learning. It's abstract, but this is usually the best sign that a candidate is staff+.
Hey Evan, thank you so much for the great content. One question, would you recommend any resources on the Design Amazon interview question? I don't normally see it being discussed on Alex Xu's books, hello interview, grokking de SDI, etc... Would be awesome seeing out tackle it!
Checkout the written breakdown for Design gopuff on our site! I think it’s really close to what you’re looking for
Thank you so much!
very good content
Hi Evan, this is great stuff!
I do have a question though regarding the 3rd API endpoint. Are you assuming that the changes were made on the remote rather than locally because it seems that the user would call the "changes" endpoint to get the changed files on the remote since a particular timestamp? It is my understanding that generally files would change locally and then you would need to sync those changes to the remote.
Happens in both directions! That api endpoint is needed to get changes from remote to local. When a local change happens, then its just the upload endpoint to sync the change.
Your videos are v. good but your mock interview is costly for certain markets (BFS can give you more return). Wish to get interviewed soon !
Please make a video around Designing a distributed login
Some interviewers really like seeing estimation numbers earlier on before going into the higher level design despite what you generally suggest we should do. Can you may be also create a video where you do some relevant capacity estimations early on and later guide the higher level design? I would love to see if there is a good way to get some numbers in during "requirements" phase
In general I’d say ask up front. Explain your reasoning and ask if they’d like you to then proceed from there. It’s pretty hard for me to think of what estimations I’d do directly up front that would inform anything. The classic bandwidth and storage is far too crude to inform in my opinion and I can already estimate based on DAU alone until later in the design.
I truly appreciate all the effort you've put into making these amazing videos. Please keep them coming. One insignificant (not important) nitpick. 50 GB @ 100Mbps = ~ 1hr 7min. I think you just forgot to convert the decimal to minutes. You have it correct in the write up, as in 1.11 hours (0.11 * 60 = 6.6 minutes).
Mental math is hard 😛
@@hello_interview tell me about it... Also not fun under the pressure of an interview 🤣
Hi Evan,
Thanks for the detailed explanation! Very helpeful!
At 40:50, you mentioned that S3 exposes multi-part upload API. Does that mean on client end we don't have to handle chunking and fingerprinting given that we use S3 multi-part?
Thanks!
You’ll still do the chunking but it will handle fingerprint checks
had few questions and suggestion->
1. When really large file is divided into chunks and upload in different s3 file path , will different pre signed urls be requested for each chunk of that large file ? will this logic be maintained in file service. If we are using trust and verify along with s3 multipart api , i guess in that case s3 would created different paths for each chunk which we also need to store in metadata , would s3 return actual file path of each chunk to client so that it can updated by file service in metadata db .
2. Also generally google drive doesn't allow to update same file , it will create a new version if a new upload is made to same path , so syncing of chunks for 50gb file that has been modified is probably not a valid usecase ?(thoughts) more valid usecase is addition of files to same folder/path .(thoughts) , which i think here would work by sync service using user_id , folder_id , timestamp , returning all metadata to client which is essentially added to folder/directory for user .
3. while responses by sync service or file service for downloads are being sent to client , i guess some pre processign would be needed to get pre signed urls from actual s3 path which is stored in ddb , which can be used by clients to donwloads .
But thanks for the video its super helpful .
Great content. Can u tell me which draing tool u are using?
This is Excalidraw!
45:45 I wonder how syncing would change if instead of folder status, it's for database writes with many writers
Do you suggest a different delivery framework for system design interviews which aren't necessarily "product"?
Topical! Was chatting about updating the site with that soon. I’d recommend very similar, but core entities and api are what may change as they could be less relevant. Instead I’d frame it as focusing on the inputs and outputs of the system more generally. And then still thinking about the data persisted
I’ll do a pure infra question next
Thanks for this amazing video! It was really informative. I had one question on Updating the Sync Service to look for any chunks that might have been updated. Once we make some edits locally (say the file size increases from 50 Mb to 70 Mb), won't all the chunk bytes shift/change which would mean we need to update all the chunks. Is my understanding correct?
If the changes are append-only (at the end of the file) we wouldn't necessarily have to re-upload the whole thing. But other types of changes might require the whole file to be re-uploaded. There are file syncing algorithms for avoiding this (e.g. the rsync algorithm), the problem is those don't work so well when using an immutable object store.
How does an event bus system clean up unnecessary work like file A changed to file B but changed back to file A again, you don’t have to walk through the events to reconstruct the folder?
If you split the files into chunks are you able to upload all chunks using the same pre-signed URL? Or do you need to generate a new URL for each chunk? How would you piece together the file from S3 when you download it through chunks?
S3 has a multi-part upload api that only requires 1 pre-signed url. Depending on if you need the chunks as chunks in S3 or not you can use that (it stitches them back together automatically). If you want to save the chunks as chunks, you need N pre-signed urls
You're vidoes are quite informative and interesting, thanks for creating those. I have a question though, the approach we take while API design, like let's go with the flow and come back and update, how is it looked down from Seniority prospective ?
Won't it look -ve for Senior/Staff candidates as it might look like "detailed thinking" is lacking or candidate isn't clear about their own thought process ?
I wouldn’t think so, no. But the key is communication. If you put something partial down and claim it’s perfect, then that’s bad. If you put something down and express that you know it’ll need to be refined and that you’ll come back to it as you learn more, then all good!
Guys, do anyone know why do we need a separate service to handle fileSync?
Thank you very much for the videos. One small nit pick. DynamoDB supports a maximum of 400KB per item and if we are storing all the chunk metadata in the item, for a 50GB file with 5 MB chunk size, assuming we need 100Bytes per chunk metadata, our item size would be around 1MB.
Good catch! True
Hi question about the chunk uploading, wouldn't uploading a big file result in huge amount of requests to update the metadata? For example just updating the status if chunk is 10mb for 50GB upload would be 5000 requests to update the chunk status. Would that have performance issue on the system when in scale?
That’s not that many in relative terms :)
one question- how do we merge chunks in order after downloading to local device? is it a good idea to keep some kind of sequence number for each chunk for a file?
Yah!
Maybe since data integrity was a non-functional requirement it would be worth mentioning some strategy for handling two clients modifying the same file concurrently - otherwise there's a high chance you end up with a corrupt file. Since you don't handle versioning, that pretty much necessitates locking.
Yup, definitely worth mentioning a resolution strategy there.
It would be really helpful if you can share which whiteboard tool you are using here.
Excalidraw.
Do a sells chatbot with multiple options please
How do we search files?
Hey Evan, when can we expect the next video?
Early next week!
@@hello_interview Thanks. Also if possible could you pls touch upon replication(multi-leader, leaderless etc.) and sharding strategies which one to use under what circumstances.
awesome, no more mouse over typing
Learning :)
can you please do video on news feed or messenger/whatsapp?
Soon! We’ve crawler next
Did we miss point on scalability completely? Since we are building Dropbox or Google Drive, it has millions of user and we would need to talk about replication and partitioning which are essential since we wont' be able to fit the data in a single database.
Definitely could have been a nonfunctional requirement and a deep dive here. TBH I find these conversations a bit sleepy. It would go something like horizontal scaling, load balancer, dynamicDB and S3 both fully managed so scale “infinitely.” Definitely smart to choose a DDB partition key, but beyond that, not the most interesting convo imo. Different if you’re designing s3 itself.
21st
OG 🤝