ฝัง
- เผยแพร่เมื่อ 8 ก.พ. 2025
- This is a solution for System Design Interview Question where you need to design a Video Streaming application like Netflix / TH-cam / Amazon Prime Video / Hotstar.
This problem has been asked by a lot of companies like Google, Uber, LinkedIn to name a few.
Summary of this video: www.codekarle....
Architecture diagram: github.com/cod...
Author: / sandeep1904
If you like this video, please help us grow by sharing this video with your friends on Facebook, connections on LinkedIn and anyone who can benefit from this.
PS: This is not the real architecture of any such platform. This is my take on how I would answer that problem.
#codekarle #systemdesign #netflixsystemdesign #system #design #interview #faang #netflix #TH-cam
Great work! Thank you for creating these.
The breadth of your videos is absolutely unmatched.
Extremely high availability...lot of earthquake everywhere....still you can see videos. It was really funny when you said "You might not want to see videos at that time, but thats a different story" 🤣🤣
Hey I am an SDE 2 at Amazon. I went through the entire video. Great content man! Very informative. Worth the length.
amazon sde-2's are overrated they don't know basic things
Is that true bro?@@Markcarleous1903
@@Markcarleous1903where do you work buddy
what was the need to tell that you are sde-2 at amazon, what was the requirement here🤔
Great video ..!! However if you are in a 45 min interview to design TH-cam like service…. Talking about tags .. talking about user sharing accounts. or ppl clicking through pagination to find the videos can be seen as diverting from the core requirement. As a candidate you need to strictly focus on how users upload, how the server processes the videos formats etc, second how videos are streamed when user clicks on a video. As focused and great the first part was… the second part was all over the place, if someone were to just see the 2nd half they might think the question was “design google search”
Very good, bud! Thank you!
Working on just internal applications at non big-tech, not dealing with such big-data or this detailed analytics, never makes you get aware of the scope/scale or all these testcases for these big platforms out there, and how they work such smart and efficiently.
Your code karle videos are amazing, Sandeep. No BS; comprehensive; pure distilled information. Thank you!!!
The best I have ever seen! Impressive work, Sandeep!
Hello Sandeep, good job. Liked your plain & no nonsense way of teaching. Also, I liked the color code of components (services, open sources etc.), that makes it easy to understand. One more thing, you speak slowly/calmly, which helps immensely, that sets the pace, while we hear we can think as well.
Awesome explanation Sir. I have not seen so much detailed and informative explanation for system design questions.
Thanks @codeKarle. Your system design videos are gems. These videos for sure will gain traction in future and will get its much deserved views and likes.
bohot badhiya Sandeep bhai - you are a combination of good looks, good brain, and good attitude
It is really useful to watch your videos to get some understanding about the system designs & not only for interviews.
I have doubts-
1) Why do we need to do the tagging for each chunk, for what purpose are we doing the tagging.
2) If ISP decides to do the caching on its side, then how are we planning to collect the statistics & how do we manage the accessibility of such cached content.
Every nicely covered. Thoroughly enjoyed. Doing a refresher in OTT system design and this was a great start!
Channel seems to be extremely underrated, amazing content
Very nice video, complete in all respects. cover lots of topics: CDN, search, recommendations, optimization - great content, and thoroughly enjoyed. Thanks for sharing the knowledge.
Thanks!! Glad that you find it useful :)
bhai, bahut hard work kiya hai apne explain krne ke liye.. thank you for your free service with knowledge
All I got to learn was that you're a big time fan if mission impossible! Just kidding.. Great Work! Appreciate all the hard work you've put in to consolidate all this 👍
Great Video. Thanks and kudos to the hard work you have put in.
I just have one request - when using an external component like Cassandra, if you could compare with other alternatives and talk about why you chose this, that will be of great help.
We have done that in some of the videos, at random places. Doing that everytime would have become repetitive.
You can check out the Databases video. There you would get to know about the alternatives that are available: th-cam.com/video/cODCpXtPHbQ/w-d-xo.html
IMO, E2E is very broad - which is super helpful! But one needs to skim through to get the basics of the components. Choice DB, caching & partitioning could have been little deeper than explaining the user login or analytics (probably a topic on it's own). But definitely very helpful overall.
Amazing video like others, thank you Sandeep! One request though, this video does not have a writeup like others. I find that very useful for making notes. Please keep up the good work!!!
I really enjoy these videos, but in a way, it's a little daunting - the analysis/explanation of all your components is taking at minimum 1 hour, where as I think most systems design interviews take place E2E within maybe 50 minutes. For example, I'm not sure if log in is something you really have to cover at all in this kind of interview, as it's mostly irrelevant and non-specific to this application. Also I might be just dumb but I feel like your system architecture design is quite higher in quality compared to other interview resources, but at the same time much more unrealistic (in my opinion). I think for me, explaining the design choices would be more helpful rather than you walking through the flow of why this architecture was set up this way. You mentioned in a comment below that it would be repetitive, but I think if you explain it in the context of each interview problem it would actually be very helpful. Thanks for all your content though, I am learning a lot!
It must be intentional - since all ppl will not have same skill - some might need it to be slow - Use the speed button to double the speed and it would suddenly turn into a 30min video
Not a bad video. Thanks for this.
Two things I learned
- The presenter's favorite movies :)
- Why I'm so frustrated by the the recommendations in any sort of systems. The "AI" thinks bc I like numbers 2 and 3 and someone likes numbers 1 and 2 and 3, -> I'm going to like the number 1 as well, however I don't like non-prime numbers, but the dumb "AI" will never notice that. And bc they copy each other's homework, all recommendation systems in the world lets me down.
Hey I went through the entire video. Great content man! Very informative. Worth the length. Can you create more video for LLD and HLD
Hey, Thank you so much all your knowledge sharing. I am able to perform very nice in all my interviews. Keep up the good work. More power to you.
Keep rocking!!!
One Suggetion. It would be be wonderful if you could break the video into chapters. The content was amazing. Thanks for sharing your knowledge.
This was really good, but would appreciate more if you could speak a more about caching and database design with tradeoffs (like relational DBs vs Cassandra), how would we cache content in CDNs, etc.
i felt that good recommendation system should be a functional requirement
Hey Sandeep, your videos are very detailed and help learn a lot. Thanks for making these videos. Please continue to make more videos as these are really awesome!!!
well explained in detail. Looking forward for videos on data structures, in java as coding language.
I appreciate all the efforts you put in sharing out this video
Very detailed and informative explanation. Thanks for all the efforts in teaching us system design.
Really awesome content and great explanation. One request from my side is please increase the video sound or use some other mic which provide great sound. Middle of video I heard very low compare to beginning or ending of the video. Thanks for the useful awesome content
For Content Processor, instead of storing chunks directly to CDN, I think transcoded chunks should be stored to Output S3 bucket and CDN can use Output Bucket as Origin server.
but wouldn't you prefer to get a file from a CDN then from a S3 bucket? What's the benefit?
@@Saurabh2816 i think he isnot saying that we should get file from S3 bucket over cdn. He is saying instead of the cdn uploader uploading the chunks to cdn, why dont we first upload to s3 and then there is an automatic sync available in aws that keeps all the cdns (cloudfronts) updated with the origin (s3 chunks) in this case. I think that would have some pros and cons
yes, that would have pros and cons
Pros
1) there will be one master source of the chunks in s3 for later retrieval if required
2) leveraging existing sync process from AWS S3 to AWS cloudfront would be super efficient
Cons
1) More redirection and complication in terms of process flow. But i dont think it is a major cons as this happens when user is not waiting for something.
49:10 I couldn't understand why you used cassandra for home page service. You mentioned that it contains information about user such as likes, dislikes etc. But doesn't this information change very often.
For example, if I watch a video on system design today, then I'd like the first row of my home page to contain another system design video. For that to happen, we have to update the Cassandra row. But I heard Cassandra is not good for updates.
I understand that it is good here because it handles reads very well and it's "always on". Cassandra scales very well with additional data. But we don't need it here, right? There won't be many new user accounts getting created every minute.
So can we use something like mysql + cache(maybe a write-back) similar to what you suggested for user service?
Super video bro! This cleared out a lot of questions and helped me understand end to end architecture of video streaming services.
Still dont know how is a video played? I mean, there are so many chunks stored in CDN server. So How does CDN organize these chunks and merge them into one whole movie? Will it merge completely before playing? Or the service just load one or several chunks at one time to give to the user? If there were any troubles during the playing, how can the service fix them?
Awesome and easily understood content!
Though it was very long but it was very informative and very detailed. Thanks for it 👍
Such great videos on system design. Hats off !!! :) :)
Absolutely brilliant. Thank you SO much for this much detail and depth.
Very very nicely explained. Thank you so much!
Loved it! though it would have been good if you could please sketch the diagram and explain each block at the same time.
Agree with this, in an interview we would be asked to create the diagram live. I would prefer if you would create the diagram on camera while explaining what each component does.
Awesome video .. very in-depth touched a lot of different things, great explanation.
Awesome video man !! Thank you so much 😍
Very detailed explanation, Great work!!!
I gave a thumbs up and watched it till the end, will this be rated as a super awesome video
your walk through is so much better than those "L8 engineer" or ex-Fang tutorials. shame that the accent is a bit thick for me
AWesome video. One feedback, the echo from the mic is making it hard to understand and if you can switch, will make it more easy
Quite detailed, thanks for ur efforts.
Great video. Got a doubt. The chunks will be stored only in the CDN ? is that the best way ? shouldn't we store the processed chunks as well ?
Amazing Video thanks for sharing this ,Can you please add Summary for all your design videos , I see you have added for a few which gives a lot of sense . Thanks Again for all the great work 👍👍👍
Thank you for the detailed explanations. What I recommend is for your videos, you can add some pictures such as showing Netflix home page during mentioning Collaborative filtering. You can add some colors to your videos to engage more. Thanks again for your effort.
Thanks! Honestly it looks like it might require a lot of editing effort, if thats not too much , we'll definitely try and do that :)
super quality content. Thanks a lot for sharing.
You said we are going to upload each chunk on the CDN, if that is true, then why are we aggregating the chunks using spark again?
This is awesome! and in great detail. Thank you for this video & Please do more on the subject. Subscribed!
Great explanation and informative content. Keep going 🙂👍
Thank you!! There is a lot more coming your way :)
Awesome video . Seamless delivery . Can the same system design be used for Audio streaming service like spotify or youtube music ?
I believe it should be usable for audio streaming. The main difference would be your bandwidth and storage requirements, because sound files are inherently different from video files. You may also consider that most users tend to listen to the same songs over and over again stored locally on their device, in contrast to TH-cam video where most users stream new videos on a constant basis while almost never downloading those videos locally. This could have implications on bandwidth as well as analytics - if you want to aggregate a user's behavior for recommendation purposes when they mostly listen to their local library (as opposed to music streaming), how do you get that information? One way would be to have the client send an event about to the server side every time a song is played or completed (this can be done asynchronously, it does not have to be sent immediately after a song is played), where Kafka could route that event to a Spark cluster, which would then be stored in a Hadoop cluster for the Recommendation Engine to consume.
Very nice video, very good explanation, thank you so much.
Can you please provide a summary for this video which has been done for several of your other presentations?
Great Content. Super helpful
You are awesome man. More power to you :D
Thanks!! Glad that you liked it :)
for predicting what you would watch tomorrow, is it ok to have the client occupy bandwith and user device storage just for the prediction? or is there a mature solution to achieve it on the server or cdn side?
Awesome video .. Covered great depth of this difficult topic ..
Before even uploadign to S3 - dont you think that content processor should first check for filteration lets say if its against the policy its not worth to upload on S3
How does the inter-service communication is handled in this scenarios, Does the requests goes through load balancer for every service to communicate since I am considering multiple instances of one or more services may be running at one point of time.
when you say local CDN 1, 2,3 at 58:50, do you mean different servers, server1, server2, serve3 within the same local CDN?
Should we use any caching for recommendation (in home page service) as well ?
Great video.
Beautiful explaination
I have a query on the upload service, i.e how are we handling the case when let's say some of the process in the content processor fails , are we maintaining state of each chunk and in case how are we handling which chunk needs to be retried?
Superb !
this video is really helpful
One ques here - why did you introduce apache spark streaming between kafka and hadoop cluster in this design? Couldn't hadoop cluster directly consume from kafka and itself perform batch processing as opposed to spark consuming all data and then sending to hadoop in batches?
Hello, I have a question - What happens in content filtering stage ( piracy, nudtity etc ) I meant lets say any of these is present then what happens, because the user has already uploaded the video so video is already in S3 now ( at least) so we then only flag and send a notification to user but content is still available ? its not like while uploading if these things gets detected then we restrict at upload only?
why cassandra
???
Questions:
1) Do you put all the data from multiple events and sources in one Kafka queue or do you use separate Kafka queues? If it is only one KAFKA queue how do multiple consumers decide which ones they want to process ? Do they have to do that inside their code or does KAFKA provide that configuration?
2) Instead of KAFKA would Amazon SNS/SQS work in this case , if not why not?
have you found an answer to this question?
looking for the same
Kafka is having concept of topics.. Multiple events are put into various topics.. Which are subscribed by the required consumer
Kafka is not meant for parallelize processing on a message-by-message basis but a AMPQ style broker is (rabbitmq/amazonsqs). So I think the video was wrong about choosing kafka to load balance messages among the content processor
Hello Sir, I have watched all your videos. Its great, but I have noticed that you have used Cassandra quite a lot. But cassandra is not known for high consistent and reads are also slow in it. So, why did you see Cassandra to store video meta deta in youtube System design?
great video
Very nicely explained.
Why did you stop creating content? Awesome video explanations!
Great content. Thank you Sandeep.
Superb explanation
Glad that you liked it!
@@codeKarle ❤️
Do share it with your friends/colleagues if you found it good 🙂
Elastic search cluster is shared between two services, is that recommended?
Raw video is stored in s3 and when we divided the into chunks we stored these into cdn. But generally cdn has TTL , what will happen when cdn service down ? Do we need to process the raw video again or do we store these processed for HA?
Peer to Peer Protocol - used in Torrents, DC++ or any filesharing across thousands of machines.
Hi codeKarle, nice video. I'd appreciate it if you could explain more about why you made decisions. For instance why cassandra? why use an async pipeline?
Is it a good idea to just throw a message queue like kafka in between everything?
Excellent video!
Very nice presentation. I have a small question, how is video played by same user account across multiple devices at same time handled. And is it synchronized, i.e., if I watch a video on my phone for 20 min, and then login to my laptop, it still shows 20 min. I am guessing we regularly send user activity information to the backend, and so anytime we login or homepage is refreshed, it loads up the user's recent activity. What do you think ?
Great video as usual. Does live streaming of cricket matches say by hotstar or disney use the same process. I am sure there will be different challenge as we wont have enough time for transcoding , creating lower quality images and then uploading chunks to cdn. how to handle that scenario where we have to live stream with say very very near real time say not more than 10 seconds
Iam a fresher and tomorrow is my Interview of system design but after watching this video I think sytem design is too much for fresher....I cleared 1st round, In my 1st round they gave me 2sum problem and deletion of anode, they are giving me 3lpa with 1yr bond.
I liked you explanation but you should try to reduce it little and make it more bullet points centric , like you explained for cassandra .
Are we storing all the CDN url for all the chunks in the Cassandra cluster ?
Please make a video on Distributed Job Scheduler.
Amazing content!
Thanks for very informative video.
@15:09 How system handles if "content filter" or any other step fails for a chunk of video ?
As the traffic is encrypted, how ISP can cache it?
Your basics are not clear
Awesome
Really nice video goes into details of many areas. details about openconnect show efforts you have put into getting to details. Since its big video do you plan to share summary link like you did with few other videos ?