Scaling Up Prime Video | Prime Reacts
ฝัง
- เผยแพร่เมื่อ 20 ก.ย. 2024
- Recorded live on twitch, GET IN
/ theprimeagen
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Microservices are overrated, I use nano-services. Every single function and method is serverless and is elastic and beanstalky... cause Bezos needs more yachts that are... BLAZINGLY FAST! ♥
Instead of a monolithic Yacht, I use a set of microboats, tied together with cat 6 wiring. It's just as fast, and much more resilient to damage/low cost.
I use Yachtless
@@Chris-se3nc Ok, I think I'm going back to the traditional method of travelling on water.
Walking...
Blazingly fast yachts for Bezos or blazinly fast apps? Choose wisely.
@@LowIiet his yachts ... you know why? Cause they don't *rust* 😜
The main thing I got from this is even AWS engineers making things for AWS can’t even figure out how they would be billed by AWS haha.
Ex-AWS engineer...this is wildly accurate. Not to make "us" (former) seem fully incompetent, but unless you're a senior engineer (L6), but mostly higher (L7) and above you aren't connected to the financials of a newer service. I didn't fully understand the billings of AWS internally as an L5, however I intuitively knew from an engineering design when AWS resources were being leveraged in a way that seemed excessive. People thinking you needed a database to store/retrieve data that could easily fit into MBs of memory on servers with GBs. It was a common mistake to see another entire AWS service as a wheel to not re:Invent (lol, see what I did there?), when in reality it was far cheaper and very reasonably doable within the application.
So you write a micro service to hold the lightbulb and I'll use step functions to rotate the house. It'll scale horizontally, only limited by your budget.
you're hired :D
Management "We need to get view counts up."
Developer "You got it boss."
You got it wrong. They are not running this for every user, but for every live stream. It's like one inspection per live stream. Not one per user
Am I missing something? Because that wasn't at all obvious from the article
@@HasanSIM14 The first sentence of the article said it was for livestreams.
This makes much more sense
“At Prime Video, we offer thousands of live streams to our customers. To ensure that customers seamlessly receive content, Prime Video set up a tool to monitor every stream viewed by customers”
first 2 sentences of the article
yeah
this makes a lot more sense in that context
thanks @titbarros for pointing this out
Client: "Video is broken, yo"
Server: "I DON'T REMEMBER ASKING YOU A GOD DAMN THING!"
It's worth noting that they're not doing this for VOD, it's for their Live Streams (such as sports). So it's to detect issues with the video stream coming into the Prime services and understanding the various realtime transcodes that are happening, not necessarily the individual customers UI or experience.
some how this doesn't make me feel better...
I wonder what a (former) twitch dev would have to say about Amazon's video streaming approach 🤔
@@ThePrimeTimeagen I mean netflix died trying to livestream a moderately popular show...
@@guidoderam Good point, do these teams talk to each other?
Yeah it feels that the primeagen was a little too much reactive on this one 😂
Pime video 2030:
"We have sent a developer to every one of our viewers so they can pause the playback at every frame and report back any issues immediately. We care about the viewing quality of our services"
might be less expensive tbh
I really don't agree with the article calling that a monolith, it's still a video processing microservice. They just packed all the computation inside one server instead of the serverless madness. But since it doesn't include all the business logic for the final product it can't be called a monolith
Agreed
Yeah, there's monolithic and then there's the spaghetti monster monolith .. the one were all infra and apps are in a single CVS repo and said app infra is coupled to the deployment so updates to deploys breaks the 10 other teams committing to the repo (and branching in CSV .. kill me now) ... ah ... the early 2000's .. the cowboy days of the intra-web
I made this exact same conclusion in a similar video about this blogpost. Nothing about this is a "monolith", it's just a better optimized microservice.
In other words "serverless" != "microservices".
@@LimitedWard yeah that’s the other funny thing. Monolith vs microservice is not the same as serverful vs serverless.
I could have a “monolith” deployed to a serverless environment like lambda or workers.
Exactly! I think that article was a publicity stunt to make us talk about prime videos. Lol
OMG, someone took serious the how to be a -10x engineer guide.
The video title is misleading it should be "Roasting Amazon Prime as a Netflix engineer."
As a millionaire 😂
Amazon is too greedy to let you watch the stream on your own, so it sends the stream back to also watch it with you. Prime video: you are never watching it alone!
Someone said "AWS certified architect type solution" lmaoo
imagine having one of the worlds thickest infra but they utilize it like this. i mean i respect the courage on the self report
they just followed the -10x developer way!
WTF, this architecture is batsh*t crazy. Somehow, I get the feeling it was merely an exercise on pushing AWS serverless infrastructure, but honestly anyone with a brain (or at least with moderate web / networking experience) could have told them what they were doing was ridiculous. Just imagine the cost to run this monstrous thing.
You're the content creator I always wanted, keep it up :)
You’re my current revelation. I stumbled across your videos three days ago and I can’t get enough. I’m laughing too hard
:) ty ty
really great to get a netflix engineer's perspective on this!
its crae crae to be multi-streaming video and audio...
@@ThePrimeTimeagen cræ
Processing video on Amazon AWS sounds like an operation that would bankrupt any company except Amazon.
So they built the most inefficient, convoluted, overengineered system to solve a problem that basically they created themselves because they don't understand how streaming works?
I love that you're uniquely positioned as industry experts on video streaming and reacting to this while conveniently you're a youtube personality
Succintly said!
"prime video" would be the perfect name for a parent company of all your channels 😅
*Bezos lawyers assemble!*
I don't think it is the matter of monolith vs microservices, they have made other optimisations and used different technologies e.g. VPS vs Lambda, VPS's typically perform better. If they replicated exactly what they have in a monolith this would be a different story. I feel the initial design was just bad design.
agreed. i still think how they are going about it is crae
@@ThePrimeTimeagen Yeah they seem to still be creating bad design. Also calling serverless microservices!? They are not the same thing.
@@ThePrimeTimeagen this is still a Microservice just in one container
Microservices never meant multiple servers or serverless don’t know where this common misconception coming from
It’s an Architectural Design Pattern
They’d till have their microservices same just all in one container
They also stopped hitting Async to Sync (oboist better for Real-Time analysis!
@@ko-Daegu Microservices are a distributed architecture e.g. multiple machines or containers lol
Remember when streaming was done on UDP where you don't really bother to confirm that the packet made it?
This happens when you have so much computing power AND you have nothing to do. So, you do the same thing multiple times, in parallel.
Haha prime acted exactly how i felt reading that article 😂
The “I work at Netflix btw” is always good
prime flexing his prowess as CEO (chief edging officer) in this reaction video. Move that detection code to the edge/js/browser!
Amazon's Prime app on my smart TV is slow as hell - the animation is clunky, it takes a second to respond to ANY remote control input etc. In contrast the Netflix app is really faast and fluid, with smooth animations including zooms etc. It's like Amazon is using a web browser wrapper and Netflix is native.
This screams "I am the manager/Teach lead" and this is going to be our archetecture...
Yeah, when I read the article, my first thought wasn’t that they shouldn’t have made this a microservice. I mean, the y obviously shouldn’t have, but there was a bigger issue in the over-engineering of the problem. This doesn’t sound like a solution that was created by an engineer, but like an overzealous PM os something. And I’m actually upset that they just threw it all into ECS and just said “It’s better now!” instead of thinking if there was just a more effective way to solve the problem.
and I still cannot get job at Amazon. these leecode peeps
@@voicevy3210 for real. I’ve been turned down several times by Amazon. And I know I could come up with something better than this lol.
You assume they're capable of finding a more efficient solution to this problem.
@@tokiomutex4148 I thought it would be a safe assumption with Bezos’ money and all.
This sounds like someone higher up made a high-level decision to analyze their image quality, and then the engineers scrambled to get this implemented without every thinking whether it's a good idea and pushing back. Lots of ants doing their job in their own little world without anybody ever reviewing the big picture again.
Would you risk your job to protect a rich guys quarterly bonus? What are you gonna do tell your boss his idea is stupid? No you don’t, you get paid and let your boss take responsibility for his actions.
@@PanosPitsi That's a very toxic work environment then. I wouldn't want to work at such a place.
@@anlumo1 Yes, I definitely wouldn't wanna work at a company, get paid 250k + bonuses and stock options to write and push some code with very little responsibility lol.
@@Btechdom if I hate working there, then yeah… not worth it. If I can get a job that I actually enjoy for 1/4 of that I’d probably be happy….
You are paid to do what you are told little man, if you dont like it then make your own Company or rise to manager.
Prime Video engineer: *Slaps roof of new monolith* This bad boy can fit thousands of stream in it
so many thousands
@@ThePrimeTimeagen those thousands of streams are being streamed to millions right?
I believe they meant thousands of streaming sources instead of thousands of concurrent consumers. Anyway, looking good mr Prime
The analysis is on the stream coming INTO prime video from the customer streaming their event THROUGH prime video, no every individual viewer...
2:30 exactly what i thought. also funny the step away from aws lambda/step functions and say "microservices". what now, lamda or microservices, are we talking about??? sounds like the original architecture was in the "everything is a nail" phase when they discovered their shiny new hammer == lambda step functions
It’s time to unleash CHADStack onto the world on the oldest Monolith in existence, the Mainframe!
I use megalith architecture and achieve peak maintainability. Basically all of my code is in a single file.
0:31 Wait till you find out who owns twitch...
It sounds like Prime secured enormous amount of compute and developer recources and they fear that if they did things efficiently and used only a small fraction of recources then they would lose most of it.
1 Lambda invocation per frame is too expensive?
whatta you know
I work for a huge smarttv manufact and we have a whole ec2 fleet using salt and ffmeg to analyze EVERY STREAM and tag the content type, this is later used to relate customer preferences and/or sell data to marketers... the interesting thing about it is the tv client classification mechanism works on any tv input. Watching your fav *Hub through hdmi or airplay? We can classify it (if you opted in). We dont use these point stores to alter bitrate though or correct anomalies, obviously the client is configured to downgrade as network appropriate.
i just realized that prime is named after competitor of his employer
It's strange to me that the article uses the terms "microservices" and "monolith" to refer to the architecture of this service. It's clearly not a standalone service, just a component of the larger video streaming system - it IS a microservice. They designed some tightly coupled processes, decided to force a bunch of network requests and disk I/O in between, then acted surprised when there were performance problems.
Just cause someone is smart enough to design something convoluted doesn't mean it's a smart idea to implement.
no wonder amazon video quality is so significantly better than netflix/hulu/hbo/etc
this is still insane tho lmao
Microservices was one of these management buzzwords and architects are some of the most useless people in IT. They don’t understand the complexity what loosely coupled means in most IT scenarios. They all say “it worked for Netflix and Amazon”. Well netflix are trivial compared for example to trading and real-time or near time applications. I told an architect that went onto the microservices idea at a customer that I was the lead engineer of when it was a startup. That it’s immensely complex to have to log and process transactions exactly in order in case of latency and errors (that increase with microservices) because we need to act on the moment, we are trading energy which means that actually bidding to supply or consume at a certain point changes the overal value of the market. It’s really-time or near time and missing that bid window because of latency pr microservice orchestration errors because service is down or busy is hard for communicate back in comparison to a monolith.
He went on anyways and the literally lost millions because and development, hosting and nd administrative burden went up and as I predicted the latency and the order of processing during failures(which always happens because also services need to be patched) had incurred more fines in a single year than all the previous 5 years of the monolithic trade service combined. So he was sacked! And I warned him he simply didn’t see the strengths or the weaknesses of microservices - and they are great for very simple not timing and order critical systems. As soon as there’s statement it’s a paradigm with incredible challenges
To me, this is not a message that serverless / microservices is dead. Frankly, this was an absolutely terrible architecture from the start -- just imagine how many senior engineers that this had to go through: "let's take insanely high throughput, realtime stream data and start a step function invocation (one of the slowest aws tools to exist) for each frame. Let's also add an unnecessary S3 IO call." AND THEY GAVE THE THUMBS UP ON IT!!!
And their v2 is hardly a monolith. It is super isolated in scope and functionality, but I guess since there's three logical components in it, it must be a monolith. That's so, so, so delusional. These people have never worked with real monoliths that are running full web applications with multiple teams merging code to the repo several times a day. What a joke.
Take a wrong approach. Make it a distributed system. Now you get a working disaster.
what are you doing step function?
If I'd architected something like their original set up ay my work, I'd be fired 100% lol
So does it mean amazon has ashamed itself publicly exposing their complete incompetence in streaming platform engineering in their own article where they described own mistakes in the grandiloquent beautiful speech??
I'm going to borrow their overengineering_skills in my job_security function
it's about live streams, not VOD, most people didn't seem to notice that
dude, on my LG TV .. each time I want to stream a trailer in prime video, 3/4 of it is delivered in 360p! WTF AMAZON?! :)
This article was just a plot to get prime to tell them where they’re falling short
This is NOT a monolith. This is just a microservice with a higher-level of abstraction. The previous design was so poor for the use-case.
Some things have to be separated; such as video transcoding processes from the API layer. The logic within those services should be written in a modular way, so it’s possible to detach it into separate services if necessary.
He works at Netflix btw.
What are you doing step functions 😳😳😳
The article is more about serverless than microservices.
I was laughing with u there on their design 😂 now I can’t sleep coz Im still asking myself why
Management: How do we blame the customer when their steam doesn't work?
Middle Management: Watch all the streams ourselves?
"Data transfer in memory" damn, and here I thought pointers were pointless exercises wherein one attempted to segfault as quickly as possible. Now you're telling me they actually have a use case? That use case being, don't rearrange your entire hard drive every 5 seconds?
You have to hand it to them for publicly confessing. Most of these things don't get written up even internally. Too much shame.
I love how the forbidden cursed word”Prime Video” wasn’t bleeped out AT FIRST.
And then after we clarified saying it basically it is as bad of a curse word as Voldemort 😂, and thus we began bleeping Prime Video.
Nah man serviceless computing is the future. You just type your credit card details in and they charge it. No having to deal with any code or infrastructure.
MongoDB's Atlas is actually the official cloud offering of MongoDB. Like, that's how they named it
In a way this post is great because it shows everyone that even big tech places like Amazon mess up their system design.
You mentioned not understanding why Ben did not use a 'hold' function as the layer, the problem was the hold function itself.
In other words, Ben's problem was that he was typing faster than his hand could move, his index finger and thumb would work together to hit a letter that was on a layer. But when his intentions were to hit a character that was not on a layer, his fingers would hit the next key *before* his thumb has responded and lifted from the key.
My moving this to a macro, he made it a one shot layer. His intention to hit a key not on a layer would be typed in, even though his thumb was physically pressing down the key and did not lift in time.
incredible, Wrong video.
I'm glad I wasn't the only one that WTF'ed hard at this one. In WHAT UNIVERSE was this ever a good idea?
What zero profiling and corporate hype does to a team lol
I think it would be cheaper to hire a bunch of people in Asia, make them watch the video stream and hit a button when it went wrong 😂
What I don't get is why they couldn't stay with server less but just shove all of the logic in a single lambda to remove the communication overhead. I'm curious how the price of that solution would compare
But why would you go serverless? This is a system getting called thousands of times a minute, why not build a server application for that specific purpose and let it eat all the petitions?
Prime- Because you're uniquely qualified to talk on video streaming architecture- think you could do a 5-10 minute breakdown of Prime Videos dumb architecture vs how it should have been designed. Maybe a couple graphics would be nice. Too lazy to Google around and figure it out myself.
I have the feeling this man works for netflix
Excuse me, why is this not done on video ingest? 10.000 movies at 50GB each is half a petabyte (peanuts for Amazon). Transcode each to lower bitrates on ingest and it would increase storage footprint to about 5x, but there would be no need for realtime shenanigans like this. Because the way they describe it, they do it in realtime while a person is streaming.
It’s funny how AWS team found AWS too expensive 😂
Hi, primegenrious rex, I wanted to note that they are probably doing ML detection as a process where each input probably has to be an image, hence the need to do all this, client side can't handle a vision model running even if its small (relatively). I can't think of any other way to do this process, they are probably sampling frames from stream uploading to a bucket rather than processing all of them (atleast I hope so) because I can imagine you always sample stream frames for QA and moderation purposes anyways, this would give them realtime-ish detection rate. Love your content rex man :)
Yeah, they say in the post that this is all fed to an ML model. I don't know why they need so much fuckery for that, tho.
Some PM saw the “stats for nerds” graph in yt, yadda, yadda, yadda, 7 months later: “we now have a scalable way to pester our engineers when a frame glitches a bit”
i really hope this is the masterpiece of a team of angry engineers
This whole system seems like busy work for developers some middle manager thought up to justify their job
and this is what happens when js developers try to build the backend without doing their research.
Here is the thing with AWS, at least my bill. Data is, cheap, Data out/transfer Fucken Expensive. When I read this I was also very confused.
It helps when you own the infrastructure i guess. LOL
Those of us who understand the substrate (that is, not javascript developers) understand that microservices and monoliths are... more or less exactly the same at the foundation of bits. The difference is when you apply these different architectures to different clouds, particularly AWS. AWS has too many layers of abstraction to make microservices tenable at high scale. There's an abstration between AL running on a hypervisor on bare hardware, then another abstracted hypervisor running an "instance" class on top of that, and then in the case of microservices, yet another layer of abstraction between the ec2 instance and the docker instance, and then a final layer between the docker instance and the microservice.
Cost of data transfers along backplanes aside, you're running 8 threads against a physical processor core in AWS for every *one thread in your microservice*. Now add up all the services and all the docker containers per EC2 instance, and then go learn all about context switching in linux kernels and be enlightened.
By the way netflix boy, you've really got nothing to say about the way anyone else does video streaming, going by your broken pile of horse shite.
This seems like some machine learning engineers who don't really understand streaming devices, bit rate etc. That built an awful software solution.
Ok this is the video that I didn’t know I need now in my life
Microcervices are a problem when done by companies that cannot afford Microservices.
Maybe I'm not familiar enough with how video streaming works, but I feel like bitrate alone is not enough to detect the quality of a video stream. You could have a stream with an insanely high bitrate, but if something's messed up when encoding the video, it could still be unwatchable, no? It seems like the goal here is to make sure the image you're sending to customers actually matches what they receive. Like you could have a 4K stream running at 60Mbps, but that's meaningless if all the customer sees is a purple image.
And I doubt they're running this for every single user. Likely just a representative subset of "canary" users to improve the chances of catching issues.
Serverless and microservices only work to a point.
Serverless is great when you can scale to zero. If you never scale to zero, than serverless isn't nearly as useful.
And microservices can be great when you don't needlessly split them into ever tinier pieces where nothing is really able to do any work on its own.
Microservices should still be able to be pretty close to fully contained workloads, even if not a full app feature.
Yes was waiting for this one!
Help me Step Functions, I'm stuck...
Using microservices when u don't need it is like using async when u don't need it and saying that u service works slower than its sync version (probably cause it's not a IO-bound tasks), but dumb ppl tend to use every mainstream way when it comes to choosing between mainstream ones and others
House Of Cards, the Show that got me to love Netflix.
"We can handle thousands of streams" lol
Isn't this specifically for livestreams though? I don't know much about video tech, but isn't livestreaming very different from regular videos. So they only do this to livestream videos for some reason... and the livestream videos are usually sports(?).
And with all of that Prime Video *always* lags and buffers on my [high end 2022 Sony OLED] TV, on a 600/600 FTTH connection with TV connected via Ethernet. Have not seen it happen on any other streaming service
"You see, our streaming capabilities have a preset limit. Knowing this weakness, I sent wave after wave of virtualized containers at them until they reached their limit and shut down"
- Jeff Bezos or smthg, idk.
No wonder big layoffs are happening in prime video org. This is horrible solution from what is written in the blog