I am sad I found this 3 years late, I've been developing microservices for the past 3+ years but I still learnt a lot from this. A huge belated thanks to you Josh!
Great talk, which is still relevant in 2020! Especially the practical examples, and the solutions they applied. Loved the analogy with the human body. Thnx Josh!
1:51 "how amazing the human body is and how something as simple as an act of breathing or interacting with the world is actually a pretty miraculous thing. And it’s actually an act of bravery to a certain extent. There are so many forces in the world, so many allergens and bacterial infections and various things that can cause problems for us." - watching this on Feb 12, 2021 makes me admire this guy and the company he dedicated part of his miraculous life.
This is a really informative talk on microservices, with real challenges and real solutions. The human body analogies make a lot of sense, there's a lot we can learn from nature...
It's amazing how quickly the difficulty of engineering systems becomes with scale. My first instinct seeing the architecture was "too complex" but I've never worked on anything so large, would be awesome to see first hand.
As someone who doesn't deal with this side of architecture, this is a fantastic talk. Micro-services are just a division scheme, not really an architecture. To get things really running smoothly this provides some great insight to where the real work starts.
Actually, no. What you get, mostly, from tech talks, is various tech people advertising their pet technology, in a way that appeals to other tech people. If you really want to really get into big systems, find work on one, and do thorough research - tech talks are a good start, but often leave important things out, and only show you the pictures of the car's body, without teaching you to work on the engine. To learn about cars, you go to grease monkey school, not to car shows. The reason for our industry being insanely hype driven is probably at least partially that people expect to actually learn something from tech talks, instead of considering them simple pointers to start research from.
Really fantastic talk. He used analogies well, making abstract concepts more concrete. My most important takeaway was the evolving nature of architectures. Architectures are challenged both by external factors (customers) and internal factors (operations, customer support, etc.).
one of the best talks i've seen at any conference on any subject. i'm actually using it as a real-world resource for my system design interview preparation because it's very informative in many ways (especially for a beginner in distributed systems like me :) ). thank you for making this available!
copying and learning from nature is always something beautiful. natural sciences like physics, chemistry do it since centuries. It's a good time for computer science to learn from methods, that are applied for many thousands of years.
AKA: Obfuscating Chaos - A Netflix Guide to Creating Dysfunctional Organizations Basically, a dev noticed: Hey, these two components could benefit from having isolated processes. And a manager said: Wait, could this also be used to stop Team A from complaining about consuming Team B's code? Thoughtful dev replied: Well, it depends on the specific components... Some components are fundamentally coupled, so they can't survive a partition anyway, and introducing one for the sake of team boundaries will just create more technical problems without solving the underlying social problems. Manager blinked twice, and after he rebooted he said: But they'll shut up about each others' code? Dev: I mean, technically. They'll just complain about API contracts instead -- Manager: Perfect. Dev: -- which is arguably worse, because as soon as you have a multistep process or something involving callbacks or dependency injection, you have a huge mess, and that's even if you can sidestep the CAP theorem for all of your use cases, which is a really big assump-- Manager: Yeah, yeah, whatever. So how soon can we do this?
This is the most useful presentation I've ever seen regarding large scale platforms. Shared in my org, hopefully we'll lift a lot from here for future extensions.
The way you explained microservices referring human body is mind blowing. Yes nature is the biggest Wikipedia or GitHub of everything. How much we learn or analyse or extract knowledge decides human evolution.
This is absolute perfection. I recently read a similar book, and it was absolute perfection. "Mastering AWS: A Software Engineers Guide" by Nathan Vale
When adding a single table becomes a cross-functional problem, then the architecture suite of microservices is messily constructed, making backend administration of Data, correcting, adding or removing data, a complicated task instead of a simple 5 line operation. Middle Tier and Platform Services will one day cause the entropy of the Matrix, for a matrix it is. It is an Architecture that can one day fall like dominoes in many possible zero day scenarios
A disclaimer: if you don't have to scale different parts of your application independently, don't bother to go microservice. It won't solve any of your issues. Splitting an inherently coupled monolithic application will turn it into a distributed one, which is worse by any metric. Domains should be loosely coupled with or without microservice architecture. If your monolithic application is modular enough, there should be no problem to deploy it separately whenever needed.
This man is genius... He explains things so well... It was enlightening to know that many of netflix's architecture optimizations are taken from Biology
2:33 This visualization so beautiful and mesmerizing, and you immediately get the human body analogy. The whole presentation sure was hard work, and well worth it.
Great work. I know most of all of this content (whoopdie do I know), have been building microservices (via a standard reverse proxy/load balanced REST API-first architecture) since 2007, and yet this is still an amazing watch and important to keep instilling. That's because knowledge isn't a one-time thing, or a time-oriented thing at all but rather a means to solidifying a way. I love that this content is out there and presented this well because these stories are still happening out there even 15-20 years later!
18:25 - hey guys does anyone know what software can produce this animated representation of a backend system? Would love to represent my own system this way to onboard other engineers easily.
I read the news of netflix introducing own bugs into their own system to make it more efficient long back, today I am watching how it is done. Good video with latest technological stuff like redundancy is good, bugs infection, support for multiple languages, region server failure routing etc..
Built my system on MS ASP back in 2002. I was already using something more or less like micro-services and using ASPCache(like EVCache). Competitors would have issues/shut downs while mine systems kept trucking along with very little down time. Think of worker ants in colonies. All working towards a common goal.
Man! I really feel sad watching this like I missed the technology boat to be a part of something incredible! Like my life will never mean as much as this talk.
Fun thing is what was applicable to netflix(read usage of microservices) may not be applicable to your company/startup. A lot of software developers don't realise when something is applicable and start to put this shit everywhere they can reach... Exactly this is happening with crypto and microservices right now - hype oriented development, they hear buzz they look how cool it is and instantly start to put everywhere...
Kubernetes was announced in July 2015, so I wonder why he said there's no good node orchestration tool out of the box at Netflix. Maybe they take extreme precautions when adopting new technology. Any insiders can confirm how they do it in 2020, did they build out in-house infra for that?
I'mmalso curious what they are using. But, as a hands-on user of k8s, I think he's right about the no good orchestration tool. K8s is widespread and provides an abstraction layer for the actual cloud provider you are using, preventing lock-in. But it isn't actually that good. It has higher latencies than would be ideal, by design, caused by its model of constant sync of resource definitions to actual state. It keeps failing in unexplainable ways, even without unleashing a simian army upon it - luckily, rarely catastrophically. It provides low level support but no high level functionality for devops patterns that become increasingly expected/state of the art, such as blue-green deployments, canary deployments, continuous integration and continuous delivery. Its autoscaling capabilities are similarly limited, and autoscaling is also slow. Ops people are unecessarily exposed to many internal details. It's not outright bad, it's just quite far away from ideal, or even good. Hashicorp's Nomad is a much better solution, technically - it does less, but what if does it does better, and it lets you integrate things that solve additional problems easier. And it is more flexible in what it allows you to do - by far. These two aspects allow you to build infrastructure with a lot less cluttering, which is easier to analyze and reason abou, where coupling between different systems is lower. Unfortunately it's not as wide spread, and not as frequently available as a service. And it doesn't have a similarly strong marketing. But marketing alone doesn't make k8s good.
The Theta Network is going to revolutionize how CDNs deliver content. It's a decentralized CDN network consisting of millions of user's smart devices acting as edge nodes that cache, encode, and stream video to those who need it... taking the work load off of the centralized CDNs (Netflix, TH-cam, etc). This not only saves these centralized CDNs money in operational costs, but also improves the speed and quality of the video content by eliminating the "last mile problem".
I would take this talk more seriously if Netflix wasn't a wildly chaotic mess where you need to log in on a specific platform to modify your subtitle options.
On the topic of client libraries. A lot of recent talk argues that we should keep things DRY within a microservice bounded context, but not between bounded contexts. This introduces coupling and client libraries are one example. In the .NET world these shared client libraries can also lead to some nasty dependency versioning hell conflicts. I assume this issue is the same in the Java world, but I have no first hand experience. Oh yep, he mentions it at 21:07.
I read the same thing and am a bit confused by that as it seems to introduce a degree of coupling which in my limited studying seems to be something to avoid.
For shared client libraries. What about requiring each service to make their own wrapper classes/library that only exposes the parts they need. Then, to make it safer against changes, they also create their own unit-tests for their wrapper classes? Or if they use almost the entire client library they could just create their own set of unit-tests for the original library and skip the wrapper. When a shared client library is changed its test runner runs all the different wrapper library unit-tests. Then if anything brakes they can just contact the team responsible for the service owning the broken tests and work out a solution together.
How do you solve API versioning and transitive dependency conflicts in the same library? He explained pretty well what the problems with shared libraries is.
What do you do when your business plan is, essentially, the sale of CPU cycles? You offer products that use as many of them as possible to achieve as little as minimally required. Enter microservices. Genius.
Well, this is a teory of experience, i think no one can present outspoken this without any extremely problem solved. If Netflix using Amazone then Amazone become rich because web services, i have self confidence for more focus diving into microservices world.
wow thank you for this clear explanation of complex arcatechture. it confirms my theory of possible manipulation to the market demand rather convert manipulation for the highest bidder. just my opinion. what is your opinion?
Can't express how high quality this talk is. Wonder if he was a teacher before!
Yah I was wondering the same thing . Very top tier presentation and speaker
When you are the lead developer for a large org, it is your job to be an educator at a high level.
Youre full of shit.
You have no idea of any of this.
I would say this is a perfect talk that describes how to writer microlyth and how NOT to design microservices based arch
@@ThunderAppeal Yeah thats why I have come here to learn!
One of the best talks I have seen on building systems of high scale, high availability, failure resilient and agility in undergoing change.
One of the best technical talks I've ever seen.
I am sad I found this 3 years late, I've been developing microservices for the past 3+ years but I still learnt a lot from this. A huge belated thanks to you Josh!
And I found this 3 years later you commented
Great talk, which is still relevant in 2020! Especially the practical examples, and the solutions they applied. Loved the analogy with the human body. Thnx Josh!
Such a fantastic metaphor to communicate the intent of microservices. As a younger programmer, that really resonated with me.
If you are interested in microservices content you can check our collection on InfoQ. www.infoq.com/microservices
1:51 "how amazing the human body is and how something as simple as an act of breathing or interacting with the world is actually a pretty miraculous thing. And it’s actually an act of bravery to a certain extent. There are so many forces in the world, so many allergens and bacterial infections and various things that can cause problems for us." - watching this on Feb 12, 2021 makes me admire this guy and the company he dedicated part of his miraculous life.
Watched again 3 years later and it's still so relevant.
it doesn't get old
One of, if not the most comprehensive and analogous talks on the subject with relevant, real world stories from someone who has lived it. Thank you.
This is a really informative talk on microservices, with real challenges and real solutions. The human body analogies make a lot of sense, there's a lot we can learn from nature...
Note to self: Write down summary starting from the 51:25.
P.S. I watched the whole thing and absolutely loved it.
in case you didn't write it down where you could find it
One of the best tech talks I've ever watched.
Happy you like it. You can find more on www.infoq.com/
It's amazing how quickly the difficulty of engineering systems becomes with scale. My first instinct seeing the architecture was "too complex" but I've never worked on anything so large, would be awesome to see first hand.
As someone who doesn't deal with this side of architecture, this is a fantastic talk. Micro-services are just a division scheme, not really an architecture. To get things really running smoothly this provides some great insight to where the real work starts.
Few years into corporate job and I had forgotten how many things are there in Computer Science. This talk brought me back to college.
So Informative. Tech talks are the best way to get insight into how huge applications are built, complexities involved, challenges and solutions.
Actually, no. What you get, mostly, from tech talks, is various tech people advertising their pet technology, in a way that appeals to other tech people. If you really want to really get into big systems, find work on one, and do thorough research - tech talks are a good start, but often leave important things out, and only show you the pictures of the car's body, without teaching you to work on the engine. To learn about cars, you go to grease monkey school, not to car shows. The reason for our industry being insanely hype driven is probably at least partially that people expect to actually learn something from tech talks, instead of considering them simple pointers to start research from.
This should be mandatory viewing for anyone working in a microservice architecture.
Speaker: "Breathing is a miraculous act of bravery."
2020: "Challenge accepted."
he is breathing in micro doses so that he can serve a continuous presentation
Really fantastic talk. He used analogies well, making abstract concepts more concrete. My most important takeaway was the evolving nature of architectures. Architectures are challenged both by external factors (customers) and internal factors (operations, customer support, etc.).
This talk is amazing. I'm far from being a system architect but this gives me a bigger perspective on how amazing systems like Netflix is built.
what I've learnt important from this video is keeping perfecting your tools & architecture.
His analogies to the human body make great sense. Awesome video. Thank you.
@Newtube Exactly.
They do. But a regular monolithic application can also be analogous to a living organism.
I just realized that on the globe we have more than 2 millions people that have interest in microservice architecture
I understand that Netflix is not only "NETFLIX" now.
Thanks for your team, too. Amazing topic.
"The structure of any system designed by an organization is isomorphic to the structure of the organization." Melving E.Conway
one of the best talks i've seen at any conference on any subject.
i'm actually using it as a real-world resource for my system design interview preparation because it's very informative in many ways (especially for a beginner in distributed systems like me :) ). thank you for making this available!
This is one of the best videos to get insights into micro services architecture.
Such valuable insight from an articulate and knowledgeable insider. TYVM
copying and learning from nature is always something beautiful. natural sciences like physics, chemistry do it since centuries. It's a good time for computer science to learn from methods, that are applied for many thousands of years.
AKA: Obfuscating Chaos - A Netflix Guide to Creating Dysfunctional Organizations
Basically, a dev noticed: Hey, these two components could benefit from having isolated processes.
And a manager said: Wait, could this also be used to stop Team A from complaining about consuming Team B's code?
Thoughtful dev replied: Well, it depends on the specific components... Some components are fundamentally coupled, so they can't survive a partition anyway, and introducing one for the sake of team boundaries will just create more technical problems without solving the underlying social problems.
Manager blinked twice, and after he rebooted he said: But they'll shut up about each others' code?
Dev: I mean, technically. They'll just complain about API contracts instead --
Manager: Perfect.
Dev: -- which is arguably worse, because as soon as you have a multistep process or something involving callbacks or dependency injection, you have a huge mess, and that's even if you can sidestep the CAP theorem for all of your use cases, which is a really big assump--
Manager: Yeah, yeah, whatever. So how soon can we do this?
I could watch an entire season of this stuff. Way better than the standard Netflix products.
This is the most useful presentation I've ever seen regarding large scale platforms. Shared in my org, hopefully we'll lift a lot from here for future extensions.
this beast is *ONLY* working for netflix for the last +20 years, that's pretty amazing! thank you for this video, great presentation!
Excellent talk. Envious of your delivery! Thank you!
Thats crazy i never even thought about all this stuff before, because i use pirate bay.
The way you explained microservices referring human body is mind blowing. Yes nature is the biggest Wikipedia or GitHub of everything. How much we learn or analyse or extract knowledge decides human evolution.
Using organization as examples are really self-explanatory. Brilliant talk!
Thanks for reminding me I don't have to think about breathing.
very good and a very open talk, no generalities - really an insider's look into these challenges - awesome!
Great video! thank you for putting this on youtube
I would love to just sit in a room with this guy and listen. Such a wealth of knowledge and experience.
Simple and valueable. A very nice and practical talk on Microservices
Awesome stuff, after watching this in 2020, i got myself thinking what changed in the architecture to this actual year.
This is absolute perfection. I recently read a similar book, and it was absolute perfection. "Mastering AWS: A Software Engineers Guide" by Nathan Vale
nicely explained .. I wish Netflix gets all such videos on Netflix and show us how do they architect and design their mammoth system
Thanks, this analogy, and learning is still happening and needs to be implemented in multiple applications/organization
Great talk and hope i will understand much more years later.
When adding a single table becomes a cross-functional problem, then the architecture suite of microservices is messily constructed, making backend administration of Data, correcting, adding or removing data, a complicated task instead of a simple 5 line operation. Middle Tier and Platform Services will one day cause the entropy of the Matrix, for a matrix it is. It is an Architecture that can one day fall like dominoes in many possible zero day scenarios
A disclaimer: if you don't have to scale different parts of your application independently, don't bother to go microservice. It won't solve any of your issues. Splitting an inherently coupled monolithic application will turn it into a distributed one, which is worse by any metric.
Domains should be loosely coupled with or without microservice architecture. If your monolithic application is modular enough, there should be no problem to deploy it separately whenever needed.
This man is genius... He explains things so well... It was enlightening to know that many of netflix's architecture optimizations are taken from Biology
Uncle Bob always talks with so much clarity. He has a deep understanding of all the topics he presents.
This is a really great talk to understand problems and solutions with microservice oriented archtecture at a large scale.
2:33 This visualization so beautiful and mesmerizing, and you immediately get the human body analogy. The whole presentation sure was hard work, and well worth it.
it's not accurate though, it shows that microsevices are one directional, which is not true.
Great work. I know most of all of this content (whoopdie do I know), have been building microservices (via a standard reverse proxy/load balanced REST API-first architecture) since 2007, and yet this is still an amazing watch and important to keep instilling. That's because knowledge isn't a one-time thing, or a time-oriented thing at all but rather a means to solidifying a way. I love that this content is out there and presented this well because these stories are still happening out there even 15-20 years later!
Turning Occam's Razor into a lecture. Well done.
We need more of these kind of lectures. Spot on sir please do contribute more.
probably one of the best guides on youtube for microservice arch
"It started out innocently enough with Python..."
So many things do...🤣
Innocuously*
scrapy comes to mind.....
@@lour.222 hahaha...
Absolutely Awesome. Love the way how Josh presented. Appreciate you for sharing the knowledge.
really good talk still in 2024 if you're new to microservices
18:25 - hey guys does anyone know what software can produce this animated representation of a backend system? Would love to represent my own system this way to onboard other engineers easily.
I read the news of netflix introducing own bugs into their own system to make it more efficient long back, today I am watching how it is done. Good video with latest technological stuff like redundancy is good, bugs infection, support for multiple languages, region server failure routing etc..
Very good stuff Josh you are a smart man and a great presenter of complex information
How can anyone even give a thumbs down to this? Very interesting and informative video.
Built my system on MS ASP back in 2002. I was already using something more or less like micro-services and using ASPCache(like EVCache). Competitors would have issues/shut downs while mine systems kept trucking along with very little down time. Think of worker ants in colonies. All working towards a common goal.
Its one of the best story regarding microservices implementation. Cool
presentation is great, hands down.
Awesome presentation!! Top class!
Man! I really feel sad watching this like I missed the technology boat to be a part of something incredible! Like my life will never mean as much as this talk.
Fun thing is what was applicable to netflix(read usage of microservices) may not be applicable to your company/startup. A lot of software developers don't realise when something is applicable and start to put this shit everywhere they can reach... Exactly this is happening with crypto and microservices right now - hype oriented development, they hear buzz they look how cool it is and instantly start to put everywhere...
Kubernetes was announced in July 2015, so I wonder why he said there's no good node orchestration tool out of the box at Netflix. Maybe they take extreme precautions when adopting new technology. Any insiders can confirm how they do it in 2020, did they build out in-house infra for that?
I'mmalso curious what they are using. But, as a hands-on user of k8s, I think he's right about the no good orchestration tool. K8s is widespread and provides an abstraction layer for the actual cloud provider you are using, preventing lock-in. But it isn't actually that good. It has higher latencies than would be ideal, by design, caused by its model of constant sync of resource definitions to actual state. It keeps failing in unexplainable ways, even without unleashing a simian army upon it - luckily, rarely catastrophically. It provides low level support but no high level functionality for devops patterns that become increasingly expected/state of the art, such as blue-green deployments, canary deployments, continuous integration and continuous delivery. Its autoscaling capabilities are similarly limited, and autoscaling is also slow. Ops people are unecessarily exposed to many internal details. It's not outright bad, it's just quite far away from ideal, or even good. Hashicorp's Nomad is a much better solution, technically - it does less, but what if does it does better, and it lets you integrate things that solve additional problems easier. And it is more flexible in what it allows you to do - by far. These two aspects allow you to build infrastructure with a lot less cluttering, which is easier to analyze and reason abou, where coupling between different systems is lower. Unfortunately it's not as wide spread, and not as frequently available as a service. And it doesn't have a similarly strong marketing. But marketing alone doesn't make k8s good.
Thanks Josh, It was an insightful talk, Thanks for sharing how netflix evolved over period of time.
32:17 When you're a sports cameraman doing a conference for some side cash
The Theta Network is going to revolutionize how CDNs deliver content. It's a decentralized CDN network consisting of millions of user's smart devices acting as edge nodes that cache, encode, and stream video to those who need it... taking the work load off of the centralized CDNs (Netflix, TH-cam, etc). This not only saves these centralized CDNs money in operational costs, but also improves the speed and quality of the video content by eliminating the "last mile problem".
Amazing explanation.
I would take this talk more seriously if Netflix wasn't a wildly chaotic mess where you need to log in on a specific platform to modify your subtitle options.
One of the best tech talks period
I liked the 2017 version of netflix so much better, but thanks for this, fantastic presentation.
Thanks Josh...very nice presentation!
Great presentation and nice analogies.
Amazing talk! Great work Josh keep it up.
Awesome post.. really worth viewing..
On the topic of client libraries. A lot of recent talk argues that we should keep things DRY within a microservice bounded context, but not between bounded contexts. This introduces coupling and client libraries are one example. In the .NET world these shared client libraries can also lead to some nasty dependency versioning hell conflicts. I assume this issue is the same in the Java world, but I have no first hand experience. Oh yep, he mentions it at 21:07.
I read the same thing and am a bit confused by that as it seems to introduce a degree of coupling which in my limited studying seems to be something to avoid.
Yep, similar issues in Java world
For shared client libraries. What about requiring each service to make their own wrapper classes/library that only exposes the parts they need. Then, to make it safer against changes, they also create their own unit-tests for their wrapper classes? Or if they use almost the entire client library they could just create their own set of unit-tests for the original library and skip the wrapper. When a shared client library is changed its test runner runs all the different wrapper library unit-tests. Then if anything brakes they can just contact the team responsible for the service owning the broken tests and work out a solution together.
How do you solve API versioning and transitive dependency conflicts in the same library? He explained pretty well what the problems with shared libraries is.
good stuff
Awesome delivery. This content is divine
DR / Availability are always going to be relevant...
I like the description of Microservices from the dude in the audience more than Martin Fowlers
What do you do when your business plan is, essentially, the sale of CPU cycles? You offer products that use as many of them as possible to achieve as little as minimally required. Enter microservices. Genius.
This Lecture was awesome.
Excellent Talk !
Well, this is a teory of experience, i think no one can present outspoken this without any extremely problem solved.
If Netflix using Amazone then Amazone become rich because web services, i have self confidence for more focus diving into microservices world.
wow thank you for this clear explanation of complex arcatechture. it confirms my theory of possible manipulation to the market demand rather convert manipulation for the highest bidder. just my opinion. what is your opinion?
Surprised by the COVID cameo at 3:00 - always feels weird to see celebrities before fame
But what do you do when Omega Star doesn't support ISO timestamps?
It means Galactus won't be able to find the birthday provider
13:23 What a background to choose for 2016
levelsetting, so that we have same vocabulary.
challenges and solutions
relationship between organisation and architecture
Amazing stuff..Thanks for the video :)
Superb video - thanks!
2:04: "There are so many forces in the world, so many...bacterial infections and various things that can cause problems for us" - Ahead of its time
And then bam, surprised COVID image at 3:00 lmao
@@DodaGarcia Loom'nati!
16:29 The vaccine doses :P
This went to spooky level