you are so charming while explaining smtg. im not on the system design stuffs but watching your videos and started to learn smtg about. thanks a lot:))))
This is an old video but I must say that it's a terrible idea to include any sort of file storage or queues in the live publishing path. What Gaurav's done here is taken a simplistic "design youtube video upload" architecture and labeled it as live streaming. Generally, with live video, you'd want to minimize end to end latency, and you'd want to re-stream transcoded frames to multiple replicas. File storage, queues, and async transcoder tasks are all antithetical to minimizing latency. As for protocol, UDP would be a great choice streaming the video to users (no need to maintain thousands of TCP connections). You can deal with out of order packets and packet drop in a live setting. When learning system design from these videos, we should always be skeptical of what's being said. No one person is an expert in all domains, or has ideas that are 100% good.
Gaurav has some ideas and maybe good for practice for like interviews but in real world there are a lot of complexities to deliver such service in a scalable matter. also Guarav rushed to publish this I think, he could have added some points on CDN, error handling, different codecs, latency, and others issues related to live streaming, anyway go take a look at people working in streaming company tech presentations and you learn a lot and also try to implement it yourselve.
We had stream outage in ipl finals 2022! Well explained! 👏 processing at transformer service will make delay isn't? So we see a delay as opposed to satellite ?
Hey Gaurav, great video again! Thank you. Do you think you'll ever do a video on System Integration? How do you get System A to talk to System B? Or how would you migrate a database from one system into another? These are common occurrences in mergers and acquisitions. I'd love to hear your thoughts on it. Thank you for the consideration.
Hey Gaurav, just a suggestion- please try to avoid zoom in, zoom out in videos, gets a bit frustrating at times. Very good content though, thanks a lot!!
How to scale queue and configure partition in this case? We cannot predetermine how much live streams will happen on platform and accordingly make partitions. Can you highlight this?
Question is how will subscribers like tablet , mobiles read from the queue ? kafka wont be running on these or any other messaging queues , correct ? will it be a websocket call ?webhooks or httpstreaming ?
Hey Gaurav I want to buy out your course but I don't see any where over portal also coverage of code and practical Implementation . Can you or your team will help me on that part?
Thank you for all the great contents Gaurav. I've learnt a lot from your videos. I have a question, instead of using a queue, can't the stream of video be ingested in some streaming platform like AWS Firehose transformation done there and the transformed data be pushed to say S3 bucket to be ingested by the intended app.
If we are storing every video fragment in a DFS, then after it's been converted the first time, do we need to send a message to the second queue informing that the worker completed its job? Ie: 1. We convert a video fragment at ts 1m20s - 1m30s using a worker. 2. That fragment is stored in the DFS. 3. The worker pushes an event to the queue saying it's finished its work. 4. The subscriber grabs that message and then the client is given the video fragment. 5. A second user steams those exact same 10 seconds. We don't need that fragment reprocessed. We can just grab it from the DFS. So the current architecture makes us reprocess every fragment repeatedly every time. So instead of having an event sent that the processing job is done, shouldn't we first check to see if that fragment even exists in the DFS, and use it if it does, rather than repeatedly reprocessing it?
The message queue system adds temporary storage by producer and consumers. That’s going to add to ur hand waving latency. This system design is not suitable for live video but works for VOD. I’d be shocked and sad if anyone is using this design for live steaming.
@@pankajchaudhari2588 Time to get shocked and sad: th-cam.com/video/cd40-l-6EmI/w-d-xo.html Watch from minute 42:00 for the relevant bits. Please don't assume that message queues are slow. Many real-time production systems use them with less than 1 ms of latency.
Hmmmm…I did a quick 1.5x and skip ahead of that video but didn’t find anything showing their video processing pipeline to be what u showed. Anyhow…if u have anything that shows that this is what Hotstar does, I’d be grateful to check it out. If not, no sweat. I am simply speaking from my experience of having been an architect for a live transcoding product.
Hi Gaurav, what would best suite to be the transformation service in an actual setup? I think a serverless computing service like AWS Lambda would work well right?
you are so charming while explaining smtg. im not on the system design stuffs but watching your videos and started to learn smtg about. thanks a lot:))))
Thank you 😁
@@deniz-of5nz simp
can't thank you enough for your videos and the amount of information the contains.
Gaurav mentioned backpressure without actually mentioning backpressure. Noice!😎
Love your content! Keep it up. If possible make videos in LLD
Thank you!
This video is above me, but I enjoy these technical videos just to get an idea on what's going on via the backend.
This is an old video but I must say that it's a terrible idea to include any sort of file storage or queues in the live publishing path. What Gaurav's done here is taken a simplistic "design youtube video upload" architecture and labeled it as live streaming. Generally, with live video, you'd want to minimize end to end latency, and you'd want to re-stream transcoded frames to multiple replicas. File storage, queues, and async transcoder tasks are all antithetical to minimizing latency. As for protocol, UDP would be a great choice streaming the video to users (no need to maintain thousands of TCP connections). You can deal with out of order packets and packet drop in a live setting. When learning system design from these videos, we should always be skeptical of what's being said. No one person is an expert in all domains, or has ideas that are 100% good.
I need help with streaming...can we connect
Gaurav has some ideas and maybe good for practice for like interviews but in real world there are a lot of complexities to deliver such service in a scalable matter.
also Guarav rushed to publish this I think, he could have added some points on CDN, error handling, different codecs, latency, and others issues related to live streaming, anyway go take a look at people working in streaming company tech presentations and you learn a lot and also try to implement it yourselve.
recently found your content. very well versed! keep up the good work!
We had stream outage in ipl finals 2022! Well explained! 👏 processing at transformer service will make delay isn't? So we see a delay as opposed to satellite ?
Hi! Can you make a video on live streaming using RTMP vs HLS?
Hey Gaurav, great video again! Thank you. Do you think you'll ever do a video on System Integration? How do you get System A to talk to System B? Or how would you migrate a database from one system into another? These are common occurrences in mergers and acquisitions. I'd love to hear your thoughts on it. Thank you for the consideration.
Hey Gaurav, just a suggestion- please try to avoid zoom in, zoom out in videos, gets a bit frustrating at times. Very good content though, thanks a lot!!
Hey, I want that load balancer hoodie. where do i get it ? BTW big fan of yours
How to scale queue and configure partition in this case?
We cannot predetermine how much live streams will happen on platform and accordingly make partitions.
Can you highlight this?
Question is how will subscribers like tablet , mobiles read from the queue ? kafka wont be running on these or any other messaging queues , correct ? will it be a websocket call ?webhooks or httpstreaming ?
Thanks for the vid!
Thank you!
I need a suggestion from you. Do u know how can i get the subscription package for streaming for website?
What is the frequency of tasks done by workers now !
Any mock interview series wich match the maang level.
Hi, can you explain what is purpose of DFS here? Thank you
for each stream or match, do we have a different queue ? I guess by adding queue we are introducing the delay .
Hey Gaurav I want to buy out your course but I don't see any where over portal also coverage of code and practical Implementation . Can you or your team will help me on that part?
Could you send us the query at contact@interviewready.io, or on the chat of the website? We'll take it forward from there 👍
Great Content
You are the best ❤️
Hi Gaurav , Great video. I emailed you a question can you please look into and respond when possible
Thank you for all the great contents Gaurav. I've learnt a lot from your videos. I have a question, instead of using a queue, can't the stream of video be ingested in some streaming platform like AWS Firehose transformation done there and the transformed data be pushed to say S3 bucket to be ingested by the intended app.
Yes, you can do that too.
greatly explained
If we are storing every video fragment in a DFS, then after it's been converted the first time, do we need to send a message to the second queue informing that the worker completed its job?
Ie:
1. We convert a video fragment at ts 1m20s - 1m30s using a worker.
2. That fragment is stored in the DFS.
3. The worker pushes an event to the queue saying it's finished its work.
4. The subscriber grabs that message and then the client is given the video fragment.
5. A second user steams those exact same 10 seconds. We don't need that fragment reprocessed. We can just grab it from the DFS.
So the current architecture makes us reprocess every fragment repeatedly every time. So instead of having an event sent that the processing job is done, shouldn't we first check to see if that fragment even exists in the DFS, and use it if it does, rather than repeatedly reprocessing it?
👏👏👏
where is cdn?
Are you in the southern hemisphere?
Nope. India 😁
@@gkcs if you feeling too cold, please do a LLD.😁
By Live Streaming, I'm assuming that it is Real time stream to users. How is this system really Real Time as it will be very slow and slugging?
What makes you think it'll be slow and slugging?
The message queue system adds temporary storage by producer and consumers. That’s going to add to ur hand waving latency.
This system design is not suitable for live video but works for VOD. I’d be shocked and sad if anyone is using this design for live steaming.
@@pankajchaudhari2588 Time to get shocked and sad:
th-cam.com/video/cd40-l-6EmI/w-d-xo.html
Watch from minute 42:00 for the relevant bits.
Please don't assume that message queues are slow. Many real-time production systems use them with less than 1 ms of latency.
@@gkcs hahahah…good one. Now I am sad
Hmmmm…I did a quick 1.5x and skip ahead of that video but didn’t find anything showing their video processing pipeline to be what u showed.
Anyhow…if u have anything that shows that this is what Hotstar does, I’d be grateful to check it out. If not, no sweat.
I am simply speaking from my experience of having been an architect for a live transcoding product.
Hi Gaurav, what would best suite to be the transformation service in an actual setup? I think a serverless computing service like AWS Lambda would work well right?
You are the best
would love to see some practical example at some point, instead of all these whiteboard drawings. nonetheless, great explanation.
Not clear
YOU THE BEST!
Great content++
This was oversimplified, I don't think this will help in sys design interview, no CDN, no cache, nothin.
@@HR-ee6zm understandably
More on how Hotstar works here:
www.scylladb.com/2021/02/11/disney-hotstar-powered-by-scylla-cloud/