When I was in computer science school twenty years ago, we were taught that "silos are bad" which microservices are based upon silos. I think the pragmatic real world answer is "it depends" as Amazon is learning in this case study. Monolithic and microservices are both tools in our toolbox.
Life is a circle. Strategies are too.. 😀 We often see people moving from one to another and then finally coming back to same state. I think in the current age it’s safe to say that a decision taken is dependent on the pros, cons and budget at that point in time..
One of the main advantages of microservices is that multiple teams can focus solely on their service, making the release/deployment cycle faster, along with benefits such as independent scalability. With a monolith, cross-team dependencies will increase, and the release/deployment cycle will slow down adding scalbiltiy challenges. So, it seems like there is a trade-off between cost savings and quick deployment/maintenance.
@@RN-jo8zt Here, when you say "implement Kafka in modular", I am assuming you want to have a Kafka publisher/subscriber in a monolithic architecture. In that case, yes, we can implement it. However, we might end up having scalability problems. For example, what if we want to have 100 producers against 10 consumers for a use case, monolith forces to have same count if use cases needs this then we are good. It is always a trade-off in any distributed system design. No design is perfect. My recommendation would be that, if we are developing based on a monolithic pattern, we should ensure to have multiple domain-driven modules within it. This way, if we need to scale any of the modules independently in the future, we will be ready.
Nicely explained. Important point is that people making a buzz that entire Amazon Prime is moving away from Micro-Service architecture is wrong, its just their monitoring system, the old architecture seems over engineered to use S3, Lambda, Step functions for such high volume real-time video stream analysis and defect detection system, with the new architecture they just doing in single process to reduce the data transfer and persistence cost with improved performance as data transfer and processing happens now in-memory.
i have que can we implement kafka in a modular monolith architecture? (we have different module with separet db code and we consueme/expose via rest api.
Agreed I'm not sure this is monolithic in nature. Also microservices have to be evaluated to see if they are the best tool to use. There is a lot of variation in what comprises a microservices and often discussion of a nanoservice vs a microservices. Microservices are often broken by domain and hence can have more then one piece of functionality grouped in the service. When choosing architecture it always comes down to it depends and of the largest factors is cost.
Thanks for this excellent explanation but is this really a monolith? This service has a very specific task and also stateless which would give a significant cost saving
Yes it is. Since all the 3 functionalities are tied together. They have tweaked the current code to function in this unique fashion though. It’s good that they came out and acknowledged this. So people understand that flexibility is required and we don’t have to run behind new technology blindly
I think what op was saying is that actually these 3 bits of functionality are close enough to each other that you wouldn't really blink if they had designed it this way from the start and called it a microservice. I've certainly seen people argue before that a ms should answer a particular behaviour or funcrionality, not a task or even worse be responsible for a single entity.
From the study it is clear that this has been implemented for one functionality of error detection and smoother rendering of videos , not the total application but a contained function. So the earlier decompetition was mainly over engineering... Is my summarised understanding correct?
That’s correct gautam. Architectures evolve over time and with technology. The evolution need not be in one direction always. This is a classic example of that.
No, the initial problem wasn't in the type of architecture. It was using AWS lambdas instead of ECS when you have a steady workload. Microservice architecture is irrelevant in this case.
Is the final video which is uploaded to S3 saved permanently or is it recomputed for every client request for the same video? Or is it region-wise S3 videos are separated and reused?
The aggregated results are for future reference. Also the buckets could be regional and geographic level along with language and video category specific
Bro Monolithic it's like a rod currently I am working on the Monolithic project for small change we are doing re deployment and it will takes forever....8 hrs this bothers me with this issue we are thinking 10 times is this change is required or not...to fulfill that functionality in other ways but our hairs are falling down but no way we will found eventually or finally We need to change the code 😊
Yes bro. I can feel your pain. The missing piece in the above article is Modular Monolith. If you look at my Strangler Pattern video, i have mentioned how modular monoliths make the monoliths better than unorganized monolith we created ages ago.
I think it’s a collective operational overhead win. Monitoring service could be a small part of the overall footprint of Prime Video. So not sure how much of that will reflect in the prime cost and passed on to us. Also we don’t know if they already are in profit or loss 😃
But I think they are still using replication for the s3 🪣. Each s3 prefix link only support 5k read head and 3k write head .The only outbound rule is from s3 to cdn .I think they did use S3 replication and reading the data to CDN and return to the users with closest Starbucks locations and it still need eventdriven microservices .Overall it didn’t fully migrate to the monolith way.
11:33 Yes, totally agree. Think first instead of moving into the new fashion tech stack.
Great summarization of the Prime Video Case Study!
When I was in computer science school twenty years ago, we were taught that "silos are bad" which microservices are based upon silos. I think the pragmatic real world answer is "it depends" as Amazon is learning in this case study. Monolithic and microservices are both tools in our toolbox.
Life is a circle. Strategies are too.. 😀
We often see people moving from one to another and then finally coming back to same state. I think in the current age it’s safe to say that a decision taken is dependent on the pros, cons and budget at that point in time..
One of the main advantages of microservices is that multiple teams can focus solely on their service, making the release/deployment cycle faster, along with benefits such as independent scalability. With a monolith, cross-team dependencies will increase, and the release/deployment cycle will slow down adding scalbiltiy challenges.
So, it seems like there is a trade-off between cost savings and quick deployment/maintenance.
Rightly put together. Thanks Mavuru.
can we implement kafka in a modular monolith architecture? (we have different module with separet db code and we consueme/expose via rest api.
@@RN-jo8zt
Here, when you say "implement Kafka in modular", I am assuming you want to have a Kafka publisher/subscriber in a monolithic architecture.
In that case, yes, we can implement it. However, we might end up having scalability problems. For example, what if we want to have 100 producers against 10 consumers for a use case, monolith forces to have same count if use cases needs this then we are good.
It is always a trade-off in any distributed system design. No design is perfect.
My recommendation would be that, if we are developing based on a monolithic pattern, we should ensure to have multiple domain-driven modules within it. This way, if we need to scale any of the modules independently in the future, we will be ready.
@@mavurugangadhar7178thanks . So much buzzwords now theses days for implementing something.
Nicely explained. Important point is that people making a buzz that entire Amazon Prime is moving away from Micro-Service architecture is wrong, its just their monitoring system, the old architecture seems over engineered to use S3, Lambda, Step functions for such high volume real-time video stream analysis and defect detection system, with the new architecture they just doing in single process to reduce the data transfer and persistence cost with improved performance as data transfer and processing happens now in-memory.
i have que
can we implement kafka in a modular monolith architecture? (we have different module with separet db code and we consueme/expose via rest api.
Simplicity always wins the day.
Agreed I'm not sure this is monolithic in nature. Also microservices have to be evaluated to see if they are the best tool to use. There is a lot of variation in what comprises a microservices and often discussion of a nanoservice vs a microservices. Microservices are often broken by domain and hence can have more then one piece of functionality grouped in the service. When choosing architecture it always comes down to it depends and of the largest factors is cost.
Thanks for this excellent explanation but is this really a monolith? This service has a very specific task and also stateless which would give a significant cost saving
Yes it is. Since all the 3 functionalities are tied together. They have tweaked the current code to function in this unique fashion though. It’s good that they came out and acknowledged this. So people understand that flexibility is required and we don’t have to run behind new technology blindly
I think what op was saying is that actually these 3 bits of functionality are close enough to each other that you wouldn't really blink if they had designed it this way from the start and called it a microservice. I've certainly seen people argue before that a ms should answer a particular behaviour or funcrionality, not a task or even worse be responsible for a single entity.
From the study it is clear that this has been implemented for one functionality of error detection and smoother rendering of videos , not the total application but a contained function. So the earlier decompetition was mainly over engineering... Is my summarised understanding correct?
That’s correct gautam. Architectures evolve over time and with technology. The evolution need not be in one direction always. This is a classic example of that.
Thank you for the detailed information 🙂
Burning topic and story
Awesome Presentation Bro 🔥
Glad it’s helpful sathya
No, the initial problem wasn't in the type of architecture. It was using AWS lambdas instead of ECS when you have a steady workload.
Microservice architecture is irrelevant in this case.
LOL 😂 previously the craze was for Serverless now its going back again 😂
Is the final video which is uploaded to S3 saved permanently or is it recomputed for every client request for the same video? Or is it region-wise S3 videos are separated and reused?
The aggregated results are for future reference. Also the buckets could be regional and geographic level along with language and video category specific
Nice explanation sir
Bro Monolithic it's like a rod currently I am working on the Monolithic project for small change we are doing re deployment and it will takes forever....8 hrs this bothers me with this issue we are thinking 10 times is this change is required or not...to fulfill that functionality in other ways but our hairs are falling down but no way we will found eventually or finally We need to change the code 😊
Yes bro. I can feel your pain. The missing piece in the above article is Modular Monolith. If you look at my Strangler Pattern video, i have mentioned how modular monoliths make the monoliths better than unorganized monolith we created ages ago.
Will Amazon prime cost reduce from this
I think it’s a collective operational overhead win. Monitoring service could be a small part of the overall footprint of Prime Video. So not sure how much of that will reflect in the prime cost and passed on to us. Also we don’t know if they already are in profit or loss 😃
too many microservices are really a mess.... 😭 sometimes i think to give up my software engg job.
👍💯
1
🙂
Found in LinkedIn 😂
Ya everyone is making a post on it now. 😀
Microservice is overhyped
But I think they are still using replication for the s3 🪣. Each s3 prefix link only support 5k read head and 3k write head .The only outbound rule is from s3 to cdn .I think they did use S3 replication and reading the data to CDN and return to the users with closest Starbucks locations and it still need eventdriven microservices .Overall it didn’t fully migrate to the monolith way.