Top 5 techniques for building the worst microservice system ever - William Brander - NDC London 2023
ฝัง
- เผยแพร่เมื่อ 17 พ.ค. 2024
- This talk was recorded at NDC London. #ndclondon #ndcconferences #microservices #architecture #softwaredeveloper
Attend the next NDC conference near you:
ndcconferences.com
ndclondon.com/
Subscribe to our TH-cam channel and learn every day:
/@NDC
Check out more content on software architecture and distributed systems from William and other Particular engineers at particular.net/webinars
Microservices come with promises of scalability, reliability, and autonomy. But if everything is so rosy, how come the only success stories we hear about are at places like Netflix or Uber? I've spent countless hours working on all kinds of microservice systems to come up with the definitive top 5 tips to ensure your microservices become complete disasters. Join me on a tour of insanity through some of the worst ways to make distributed mistakes.
Check out our new channel:
NDC Clips:
@ndcclips - วิทยาศาสตร์และเทคโนโลยี
![🔴 [TRỰC TIẾP] Tứ Xuyên (Trung Quốc) VS U20 Thái Lan | Cúp VTV9 - Bình Điền 2024 | JET STUDIO](http://i.ytimg.com/vi/bguJ41PcW0c/mqdefault.jpg)
Great material and it is sad that the crowd is absolutely dead. This man deserves a better audience.
To be fair conference mics nowadays often don’t pick up audience reactions (or any noise other than the speaker’s voice really)
With all due respect, when I want to come to a lecture like this, I will come for the information, and not for mediocre jokes.
@@randomrandom7208Its not a lecture! Its a conference!
@@ironnoriboihow does that change anything?
@@ceigey-authat's good design
As an old fart in this industry I have found that 9 times out of ten using caches and queues before you distribute solves the scaling problem more effectively. Cache your db read operations and queue your writes. Also relax foreign key constraints, reduce the number of indices to ones you really need. And see if you can can denormalize tables particularly those that are used to store "time series-y" data. Also, pre-compute stuff if you can.
This is solid advice.
Yup. I would only add - 1) don't skip on the indexes you do need, (so you need to check the query plans of new searches) and 2) do whatever you can in the database i.e. don't search, filter or join in middleware; and as a corollary 3) if the "SQL join" is too slow, then putting the tables in separate microservices is only going to make it worse.
Personally, I reckon ORMs and performance make uncomfortable bedfellows - what do you reckon?
@@outwithrealitytoo ORMs and problem # 3 are closely related, since it is so easy to add fields and new entities that are related to other entities, your resultant queries can become a total mess. I am not saying it can't happen otherwise, it is just that explicitly thinking about table design is generally a good practice instead of outsourcing it to your ORM
Another reason for 3 is scan type queries, where you are retrieving a buttload of data and here is where you really need to ask yourself the following questions:
1. Do I really need this?
2. Does this need to be near real time? (If not pre-compute and store the results in some place on a scheduled basis)
3. Can this be offloaded to more column oriented storage like data warehouses ( in one place I worked we provided logistics services for SMEs and our solution was to provided a daily snapshot of all transactions which was precomputed from the warehouse and dumped as a csv file into a s3 bucket that the business owner could download from his app)
Man just adding some async operations would help. We had a function that was taking 30-60s to populate a web view. All it does is query one NoSQL collection about 20-40 times.
Could the dev have dispatched the queries asynchronously then awaited the entire pool of results? Sure! Did he? No!
Dont get me started on that NoSQL mess either. Or the sheer volume of monitoring we need to figure out wtf is going wrong in our stack.
Caches can become stale quite fast. In my system we need to fetch and update customer balances fast but the source of truth is a slow third party core banking system. We had to literally rebuild that service on our side with caches to remain in sync. But it's an imperfect solution as we are not the only consumer of balances.
I've done distributed systems for thirty years. Debugged a lot of bad systems. He's absolutely spot on. I would have liked real performance graphs.
Don't really need that. I remember an older talk where someone mentioned how fast a function call is to another DLL compared to how slow an HTTP call is, or an RPC. It doesn't take a lot of math to realize one is microseconds and the other easily goes up to dozens of milliseconds, 1000x the amount of time. You have to have a lot of work to be done in parallel, for that to make sense. Or be running out of cores or something. It really should be the exception, to do work remotely.
I mean his credit card example is an example of something that should have never been in a QUEUE to begin with. These systems comes down to the decision to use a sync call vs a queued call and when and where to handle each. It's an art and the vast majority of people that implement these systems sorta make pretty poor decisions around it.
The primary reason to enable microservices architectures is none of the reasons mentioned, its mostly to allow a big team to work on applications without causing major slowdowns during the merge process and allow them to fight fires related to there issues without impacting the rest of the eco system.
@@LanceBryantGrigg the problem is that small teams are doing microservices. I think that just doesn't make sense. If you can keep your architecture simple, keep it simple. Not everyone needs to serve 4 million simultaneous users, and not everyone has 500 people dev teams.
As somebody who is currently spending a lot of time maintaining a monolith application written in WinForms, I have to say, I think I would happily jump into the rewrite pit even if it's a horrible decision.
that's why most shops are doing it or have done it. Every now and then you get a business unit willing to go 3 or 5 times over budget to finish it. it's very alluring. and even if it fails it builds your resume on good/relevant tech
Is that you dave? Never thought I'd run into someone from work in the TH-cam comments!
Yeah, the grand rewrite in the skies!
Refactoring legacy systems is really work, but calling for a rewrite is to avoid this and presumably take shortcuts. Not only that I've more than once seen people yelling for rewrite, who have had big shares themselves on the mess that they were complaining about, it's also the question why guys who aren't craftsmen enough for a refactoring should be able, based on their skills, to create something better? Is it a hope for learning by doing, but just this time better than the tries before? Don't get me wrong, I'm aware that it sometimes seems hard enough, but easier to convince business and management of a rewrite (with all the promises) than continuously or at least frequently demand refactoring and space to improve the existing, but this is where the effort belongs and it could only be survived with skills, a well-equipped toolbelt and attitude. - But without, one will fail in legacy stuff as well as in new "rewrites".
- As you write of WinForms, maybe a new UI (technology) could be reasonable.
- Not only code mess could be a driver and must be considered for refactorings, but also gained knowledge of (business) processes. However, as long as at least parts of a SW system are happily used, that is a counter indication for a rewrite, but for refactoring.
- Yes, I confess that I have also called for rewrites in the past (maybe not as loud as others) - and I saw those approaches gracefully fail for the aforementioned reasons.
Why do people think that rewrite from scratch will solve all problems. Odds are, the same problems are likely to reappear. In fact your current monolith might well be a rewrite itself..
I much rather have to maintain an old monolith than a convoluted over distributed microservices system with non-deterministic behavior all over the place. Some bugs just become unfixable.
And every developer wants to do a rewrite.. they never respect other people's code. There's often more wisdom in there than you think. And just because it doesn't use the flashiest, or newest techniques, doesn't mean it doesn't get the job done. In fact, not using the latest fashion is often a plus! (Young programmers often fail to appreciate that because they are desperate to get the new keyword on their resumes)
I work on ERP system from a very famous European company.
And They provided solution which integrates their own cloud systems to their own on prem system.
I have to say - they ticked every single problem on all of the distributed system architecture. And they are proud of it when we raise issue with the company on these issues.
The integrations are so badly maintained, half of the time you dont get all the necessary data, and half of the time, you want to do customization to make that happen you lose support to their standard product updates.
Good to see it's industry wise practice to make systems worse by introducing more systems in architecture and not just in this company's particular product.
SAP!
@@alestar22 yupp
@@rns10 No doubs :) Had to work as perfengineer with both on-prem Hybris and therr cloud v1. Latter is horrible from performance perspective.
Well known as System, Always, Problem
This company is dying, you can know it because 1/3 of the staff is lawyers whose sole job is to either write or litigate/enforce licenses. It's a company from the suits, by the suits, for the suits, that has an utterly horrible product.
This was great, now we need the complementary talk about building the best distributed system ever.
So .dll sharing was the secret 6th technique to complete the worst system ever?
Not sure about .dll, but in case of Java, I’ve seen this pattern fail hard. After a year, you have a dependency hell inside your own code base.
@@mhandle109 yes, even if you can handle the version dependencies of JARs (DDLs?), it also couples your database to multiple applications. Now you can't update your services data model on your terms because the JAR/DLL in the search engine still accesses the old data model. You could bind that engine deployment to your service so every update automatically redeploys the engine with the latest JAR, but this opens a new can of worms (testing etc.). And at the end, it won't be the only "engine" business needs.
I have no idea man. His sense of semi-deadpan humor makes it almost impossible to tell if he's ironic or not. DLL sharing as a solution to the data duplication issue in the Search Service sounds like an absolutely terrible idea, but he keeps talking about his example as if it is actually good.
@@FudgeYeahLinusLAN this sounds like a great idea at first glance, because it’s basically DRY on system level. But in practice, after a year of corner cutting, you end up with a mess. I’ve seen that
@@dvdstelt please share an example, I can't even imagine how it can be any good
This was fun. Thank you for writing and presenting.
Yep this pretty much happened at our company :D We are currently undoing the microservice projects of our architects, moving them back into the monolith.
anyone with basic CS knowledge about distributed systems would have known that splitting things at too granular level isn't going to work. Anyone who took a parallel programming class, for example. Latency is a real thing!
Scary accurate. I guess the problem is always the hype train, when a new trend comes along previous models are always considered as trash and then everyone tries to cram everything into a new model without considering what makes sense and what dosent.
Software Engineering these days is less Engineering and more resembles the fashion industry. Let's throw out a decade of frameworks, tech, know how, computer languages, etc.. and lets follow this new guy with his inmature, have backed, have documented, unproven theory.. because why?
Well it goes nice with the new summer colors.
Maybe one day these will be actual engineering decisions. The worst part is that resisting is expensive. Because you won't find developers willing to work with old proven frameworks, they're all jumping to the new shiny stuff created by the "merchants of complexity".. (basically frameworks that only add complexity rather than solve any real need)
@@acasualviewer5861 I work in C++. Not dotnet mind you. Pure C++. And also far away from any major industrial centers. The last time we needed an extra developer it took us 15 months to find one.
Oh wow, fantastic talk! A happy combination of insight and fun! 🤩
There is one more option where a rewrite is probably a good idea, when the current one is built on a platform that is going to die and you either just shut it down or replace it, and you cannot just shut it down since its vital for the business.
Maybe it is time for relational databases
So with this dll sharing now you need to monitor all the systems that decided to use your dll/you decided they now have to use your dll and whenever you need to update something, you have to roll out all the dependents.
And if that is not enough to disabuse you of this idea, what do you do if you're a search engine service maintainer and one of the dlls you're importing starts failing? Do you roll it back? What if the database behind it is already changed?
Now you can't really have any responsibility separation in your company in between any services once they decide to share their dlls.
Good thing it's in the talk name after all...
100% agree. This dll sharing pattern is the road to a distributed monolith nightmare.
I got the same impression.
Yeah but he keeps talking about the DLL sharing stuff as if it's a good thing because there's a distinction between logical and physical boundaries. At 34:08 he explicitly says "I'm going to show you a technique that can make the previous situation better", and then he proposes the interface and then the DLL sharing.
> So with this dll sharing now you need to monitor all the systems that decided to use your dll/you decided they now have to use your dll and whenever you need to update something, you have to roll out all the dependents. And in theory, yes, this could be a problem. But there are solutions for it as well. Continous deployment tools usually have APIs to ask where a specific thing is included in the deployment and use that to trigger those deployments again. But it depends on the scenario.
With the "dll sharing", the idea is not to create a single dll and deploy it everywhere you can. They can be very specific to a single thing. The important thing is to remember that data doesn't need to go everywhere. Because sharing data is the biggest nightmare. Not the sharing of DLLs
> what do you do if you're a search engine service maintainer
What does it mean to be a search engine maintainer? Writing a search engine? Probably don't do that, use an existing tool and focus on adding business value. If you're worried about stale data; data is stale as soon as it leaves the database. When you're reading this comment, it's stale, because I might've edited it.
> and one of the dlls you're importing starts failing
That's up to the code itself, just like with any other code. Roll it back? That's a decision that code makes. The way William presented it, isn't that much different from how code works in general. It's more dividing up the code and data that's different.
@@dvdstelt
1. It's not about data, it's about ownership. Yes, you will need a bunch of continuous deployment tooling to support that and that is a lot of additional effort(you will need hand-crafted pipelines that will ensure that everything deploys at the same time and then if anything fails to, rolls everything back etc., it's a pain in the ass to implement and maintain). But that doesn't address the monitoring a bit. Everybody who participates in this dll sharing would have to be monitoring bunch of different parts of the system and every such "engine" will need to be monitored by all the teams that are sharing their dlls with it. By monitoring I mean logs and alerting.
2. Afaik there's no "search service engine" that supports easy dll plugins that will cover all of the cases any project needs. So someone will be writing it (maybe not from the ground zero) and somebody will be maintaining it for its lifespan. Ownership, that's what it is.
3. That's too generic. If I'm responsible to keep the search engine running and maybe a few of the search pages and one of the dlls I've been shared starts failing, what code would make a decision? What do I roll back? The dll itself or the services that are related to that dll? Or maybe the database also that this dll is accessing? There's no easy answer to that besides generic claims of automation. Every time this happens I would need to contact the team maintaining this dll and we would need to come up with a roll back/fix plan.
And these are the worst kind of solutions. They sound so simple when we're talking in generic terms like CI/CD and automation but they never are.
this guy seems like he'd be a hoot to have as a coworker. good jokester who knows how to satire while making great points
I loved his "imagine the result using Scrum" when showing the Indian monument....
Awesome lecture 🎉
fantastic talk, I have worked on real projects that had the same issues as described here. A lot of newbies that follow trends without actually analyzing pros/cons will continue failing on the trap.
100% true - worth watching all of it - nice work
The Engine Pattern - going from shipping data across a network to shipping binaries across it instead. Much simpler.
Monolith with extra steps
I found it's like shopify modular monolith
I think the big takeaway was what would have been the subscriber to the other services now gets to define its interface and force others to comply to that. This is just an implementation detail.
People really missed the sarcasm I made in this comment. This design is so bad I literally thought it was joke, but hey this way of thinking is a trait of every "engineer" I've ever had the displeasure to work with that lives in the C#/.net |Java/JVM ecosystem
Get the poison out of your minds. Stop coupling your functions ("methods") to your data and you don't need any of this rubbish. Reject OO. It was a mistake in the same way bubbling errors ("exceptions") were a mistake.
@@mcspud I didn't miss the sarcasm. The point you made using sarcasm was what I was responding to as I'm sure what everyone else was.
For those that work in the Java World, I recommend looking into OSGi, this is a framework that will solve the Monolith vs microsoervice concepts where you have full modularity in a monolith with full individual deployments live in a running platform, it is built upon the services concept and interface way of thinking as descibed in the end of this talk. Full speed in one machine.
He exactly described what all of the smartest guys in our company designed...
Really liked this one, cause it's funny and at the same educational!
I'm doing this rewrite as we speak 😮
Great talk, I hope to watch more from him! These are the kinds of talk that I like, with a pinch of comedy👏
This is my new mentor. Every developer / IT person should watch it.
One of the major challenges with the solution is scaling .. How do you scale the system?
In the example, even though the Product and Search Engine are two separate services scalable (they are logically separate but not physically). Scaling Search service mean, you are scaling part of product service as well (of course only search related).
well he did say don't use it everywhere saying 'William said so', and said its only applicable at times. So if your architecture fits the pattern use it, otherwise use different design pattern. But in this case i don't think there is a problem with scalability, because if you want to scale 'searching', its only makes sense to scale product search service together (in the end they are sharing resources)
Oh wait. I get it. The title of the talk should be “microservices guy reinvents code libraries” I tweeted about this exact same problem yesterday. If you cannot properly partition your code you will either build a spaghetti monolith or a spaghetti microservice. Spaghetti microservices being worse because of the added network overhead.
All these design talks should start by asking people to go back to basics and properly partition your code into chunks (leas call them… modules?) that can be used anywhere and only then consider how many services you need. Hint: only convert into services what it may need to scale. Hint: not everything needs to scale unless you are really big. Hint: except four of you, you are not really big.
This guy is incredible. Great job!
write everything in main like a real man
Great talk
Excellent stuff, with a great humor :)
Most hilarious and technically sound talk ever! Pain in the cheeks!!
For the engine pattern example, what happens when there are new business requirements like adding a price filter?
Don't feel like that solution can adapt to such requirements.
I like the verb/noun distinction - not quite functional vs object-oriented but close. By "engine" do you mean distributing a shared code-base, as you describe logically identical physically separate?
I've never written a bad system - everything I've written was the best thing I've made for that time!
Who the hell distributes systems into microservices that follows a daisy chain pattern especially one that traverses over the internet between clouds? I know it happens, but the problems are just so damn obvious. I've worked for financial institutions where they always bought external vendor apps that just added more external dependencies and horrible synchronicity issues. I've seen in both commercial and government applications where they route traffic through AWS , Azure, and GCP in a single daisy chained transaction that would take 10 microseconds, but takes instead 6 seconds, but then are simultaneously concerned at the 10ms latency difference DB queries for DR sites. I hate people sometimes.
Well, glossy magazines driven, business promoted (senior) architects with an affection for "hype stuff others do"!?!
"... I hate people sometimes." - Rocks are okay, right?! 😅
How is this different from CQRS? Are you not just trading messages for direct calls to dll's?
Got one more situation where a rewrite works: When the original was written in a hackathon five years ago and then, stupidly, was maintained without being updated
I found this presentation fairly entertaining and it had a few interesting tidbits from an informational point of view. But there are a *lot* of assertions in this presentation. None of them are supported by evidence, and only a few are supported by any kind of logical argument. Most of the support for the assertions comes in the form of a sentence or two where the argument only passes muster if the audience doesn't think about it too much and just goes "I guess that makes sense...." .
In my opinion some of the assertions are accurate... but that's sort of the problem. My mere opinion agrees with the presenters mere opinion some of the time, but that's not persuasive. It's not a "good reason" to believe something.
Creating "graphs" based on make-believe data creates a sort of veneer of legitimacy, but of course there's no actual legitimacy there. Take, for example, the form of the graph at 17:16. The presenter's position correlates with the slopes. For example, if the blue slope had been much steeper than the original green slope, then the presenter's position would not be supported by the graph. If the dotted red line had been further to the right, then the presenter's position would not be supported by the graph. So, obviously, the graph is just begging the question. It is manufactured arbitrarily by the presenter to fit the narrative. This is bad. As an industry we shouldn't be making arguments in this way.
Things improve a lot in this talk after about 20:00. Here we start to see some investigation into the logical consequences of decisions, the arguments become much more concrete, and so on.
I think the Engine pattern solution is unnecessary, though. The problem with the CQRS pub sub pattern is that it didn't adhere to the verb oriented DDD boundaries that the presenter (correctly, in my opinion) described moments before. To use the discount problem exemplified in the presentation, the problem is that the *model* of the search service was coupled to the *model* of the other services. To be completely specific, the presenter says that "we have to make the same change in the search service because now the search service doesn't have statuses that we have to worry about. The coupling still exists [...]". But this is only true *because* the search service was modelled with the idea of a "status" in the first place, whereas what it really should have been doing was modelling the use case - ie, the discount itself. This is exactly where the concept of an "anti corruption layer" between domain boundaries is meant to exist (according to Eric Evans). To solve that model transformation and remove the model coupling.
Yes this kind of presentation really needs something more than just throwing up bad patterns and problems. Not sure what the takeaway is?
@@andreas_bergstrom it is nor clear if the rule engine pattern is bad or good, I'm left with that big doubt
I think you missed the whole point about the graph. The “chart” was only used, to visually show that the throughput will go to SHIT after certain load point LoooL
@@andreas_bergstrom, the take way is to:
1. Don’t build several systems that talk to each other across the network
2. Don’t underestimate how big the rewrite will be
3. Don’t re-event the wheel by building a framework
4. Use nouns and not verbs when building services
5. Deploy component that work together on the same physical tier
@@osman3404 #4 says it's a guaranteed fail if you base your services on nouns rather than verbs. But in #5 he proves that a verb based service (Search) also doesn't work. So the takaway is to neither use nouns nor verbs I guess.
In a nutshell the interface is either defined by the subscriber or the consumer but either way there is always coupling. If search changes with the engine pattern order service needs to change. There's no way around that and in this instance maybe it works because the search interface doesn't need to change as often but this is one example. I've worked on systems that use events and commands, duplicated data, API class etc etc. There's no good solution apart from keeping it monolith unless you are certain there are practically zero dependencies so data duplication isn't a big concern.
This guy hammer the nail badly on every problem I encountered working in big system microservice environment. Ouch.
Lovely talk. Bookmarking to review periodically.
Good presentation, but the last part - assemblies sharing, means DB coupling between services. This introduces at least 2 serious problems: security (i.e., who has access to the DB) ; DB schema coupling.
That’s the point - he’s telling us it’s -not- a good thing to do
The point of the talk was to showcase 5 techniques you should NOT do. Some people are missing the point and assume the conversation was about things you would do. This is also why there were no questions at the end of the talk. The topic went way over many heads, and people did not get the humor or the sarcasm throughout the discussion.
@@redlinejoes Yeah but in the last example he's explicitly saying that his suggestions with assembly sharing is a solution for the issue with the Search Service being coupled to the duplicated data. He's basically just exchanging one type of coupling for another, and I would argue both types of coupling are equally bad.
@@FudgeYeahLinusLAN r/whoosh you missed the title and the point of the talk. I don't understand how people don't get it.
@@redlinejoes Because he's literally contradicting himself. #4 says use verbs instead of nouns. In #5 he uses a verb and creates a worse situation. Well which one is it? In effect he's saying that we should use nouns rather than verbs, but all his previous examples uses nouns and they are all bad situations. It's either that or he's saying we shouldn't use either, but then this is in condradiction to his #4 rule which is an explicit dichotomy: either verbs or nouns.
And during his #5 example, he shows the DLL sharing and first says (38:49) "I always find it interesting that when people consider this type of idea that it feels weird to them, that we're putting these things together and coupling them, but this type of distinction between logical and physical is fine when you're talking about scaling out instances, taking the same code and running it in two places"... only to immediately after say "so don't do this". And then his advice is to not conflate logical with deployment boundaries. Well which one is it? Both can't be true at the same time.
I’m not a dev and have almost no clue what I’m watching, and now I’m super confused about what is a good and bad way to do any of this 😅
You miss the sarcasm and humor. Read the title. Then ask yourself, why would anyone implement these 5 techniques for building the worst microservices system ever? It's obvious. You wouldn't. The speaker repeatedly said he's into building and discussing bad systems architecture. You wouldn't do what he's telling you. You want to learn from these mistakes, so you don't try to utilize any of these 5 techniques.
To fix the actual stated problem in the beginning would be to increase the amount of memory available (scale up) and / or to deploy more instances of the monolith behind a load balancer (scale out), problem solved :) Of course if you make money per hour (consulting) or make money by bringing in more developers (like a consultancy) microservices are amazing!
Why would having an event-driven data projection be a bad decision?
It is the best decision, it is the most correct way to implement the search engine according to that use case, completely decoupled, independent storage, completely free to implement any logic on the data. If anyone can correct me I would appreciate it very much.
He gave examples of why it is bad and mainly it relates to how the logic that typically belongs to product service or the pricing service etc. is now spread around and the search engine can become a constraining factor when it comes to things like changing how pricing works
The fun thing about consulting is I've gazed into each of these abysses lol
The search service can be done simply using monolithic though much more painless
20:25 This is not how queues should be utilized. You're supposed to leave the message in the queue if an error occurs so have it move to a dead letter queue so you can investigate and then redrive the message. Not having a good understanding of queues is not a good argument against microservices.
Agreed. Strawman fallacy it seems
This was supremely relatable
Bravo
This dude absolutely nailed it lmfao waaaaay too relatable
Having only watched the part about the Gen 2 garbage collection so far, I have already learned something new! Then it must be the case that all of Microsoft's websites are permanently having a gen 2 level garbage collection event!
Microservices should been able to answers questions by themself (having a optimized copy of referenced data), and if they propagate information it should be async, this does not solve all issue, but without this its always a distributed monolith.
Very cool
I throw gen 2 GC as error starting now.
Came across this talk today. Really good design pattern for cross-domain CRUD operations.
THIS IS the BEST Conf talk I’ve watched this year loool and I apprenticed the sarcasms too loool it’s like he knew how I build things at work lool
That was very sad to realize: it is exactly what we have in our company
Well, What about referential transparency, location transparency ?
Is there a GitHub example of the actual implementation of the last example? I fail to understand how to do this in practice.
I think it could meant the following: for any service that acts like search provider to implement related logic as separate assemblies that (by convention) could be picked up by CI/CD system and deployed onto search engine. The latter utilizes these as plugins.
Microservices are great, using the wrong patterns can cause issues, this is where experience and knowledge are key.
or... you can add a secret sauce that's been around for some time now but slowly people are finding out it's actually a good binding agent to use with microservice architecture: rules engine.
I dislike the rules engine very much. It just introduces more coupling in a place where it shouldn't be.
@@dvdstelt the idea is to use the rules engine as a coupling mechanism so that other modules doesn't need to. that's why i said it's a "binding" agent. If you do enough similar project to the one brought up by the speaker in this video, you'd start to realize most of the time where client or user need you to introduce coupling in your modules, it's a business rules requirements that often needs to be highly dynamic and fine-tuneable, that's why it's such a good fit.
Well, I do agree with the take on monoliths. Like that Indian Monolith, they can be beautiful. But also like that one, they can be useless and get to be abandoned.
The real question is why they are abandoned.
Sometimes perfectly good buildings are abandoned because of a change of religion or culture, even though the building is perfectly functional.
We had a system written in "old" .Net code. Then someone came and said: "This stuff is obsolte! Doesn't even use Web API" (nevermind that web API hadn't been invented). And proceeded to convince management to rewrite it. Even though the code was perfectly scalable, and responding to requests faster than users could consume them.
The new kid with 2 years experience decided that a "monolith" was "bad". So it was unnecessarily rewritten.
Typicial "not my code" syndrome. And typical "Zoolander engineering".
If your entire project consists out of independent modules that only interact with each other through well defined interfaces, there is no need to ever rewrite the whole thing at once. You can always rewrite a single module without having to touch any other module in the process and once done, you just swap the old for the new module. Even if you need to alter the interface, as the new module will require a different interface, your rewrite is still limited to the modules that use this specific interface, which will probably be multiple modules but still only a smaller subset of all modules. And even if you need to change the way how certain modules interact or you want to combine three modules into one or split one module into three, this still limits your rewrite to a group of modules affected by it. So at all times you are only working at a subpart of the system, keeping the rest of the system just as it is, yet if you keep going and going, eventually you will have rewritten the entire system at some point, just never as a whole.
And by modules I don't mean micro services, by modules I mean a bunch of code grouped together. Of course, you can distribute modules into different processes or to different containers or to different machines or to different data centers but that's something the code shouldn't need to care for. The code only knows the interface of a module and it will use that interface to get access to any data it requires or request any action to be performed that it wants to be done. Whether this will just call a function in the current process, send a request via RPC to another process, send a request to a micro service or calls a REST API on a server at the other end of the world is non of the code's business and subject to change on a daily basis.
"just call a function in the current process, send a request via RPC to another process, send a request to a micro service or calls a REST API on a server at the other end of the world is non of the code's business and subject to change on a daily basis."
Which translates to... "worked in dev; ops problem now."
Currently the head of a microsystems project which I took over after all the parts had been built as a proof of concept and then those tent cities were expanded into some kind of horrifying turbofavela. I'm currently balls deep in the "try to extricate the various moving parts into properly isolated modules without breaking everything" step of things. It's truly satisfying when you finally free one from the wreckage, but it is a hard fought process.
@@-Gnarlemagne Love the idea of a favela in the cloud... perhaps the "enterprise" equivalent would be more of a Kowloon Walled CIty ?
@@outwithrealitytoo a Kowloon Walled CIty made by fabric, very very flammable fabric.
The key word is "well-defined interfaces". You will change the interfaces, and so the modules
29:50 - Best advice ever (I mean *worst* advice ever) :D
I don't think rewriting from scratch is always a bad thing. There are certain times when it makes sense; for me it depends on the level of complexity and familiarity.
- If it's not a complicated system, then writing your own is relatively easy and can help you discover new ways to architect it.
- Or, if the project has just been handed to you and you're tasked with maintaining/updating it, you will probably spend as much time learning the system as you would rewriting it (and you can do both at the same time! Use the old code as a reference)
I've been in both of those situations before and have successfully managed rewrites in both cases. There are also some cases where I started a completely new codebase and eventually ported the developments back to the production code. Having a clean slate gives you agility, a freedom to experiment, which is more productive for some people (myself included)
best title ever
were the reddit screenshots from the boost app? i liked the boost app :(
it is not clear, so the engine pattern is bad or good?
It’s good lool
@@osman3404 No it isn't. He's literally proposing DLL sharing. That's an extremely bad idea.
This is an excellent talk. Worth referring back to. But I really wish it wasn't presented in its negative. I get it was done for humor, and it works in that sense, but it really makes you do mental gymnastics when you try to refer back to a specific part...
Looks like everybody forget this great advertising where people build airplaine in the air. No need for Stable Diffusion
18:25 Ariane flight V88.
who is really bad as estimating, must be a really good developer :))) that was funny and I also agree :))
These multi-lip smileys, are you from Eastern Europe?
How do you know the slaves did not claw the rock away to build that temple? I've clawed the keys away from board for 13 years!
Definitely a must-have talk
9:35 Is 2011 and pre-containers even relevant today? I don't see how "stop-the-world" GC issues would cause performance issues for any API today as there would be more than 1 instance running...
Of course it is
True that it's mitigated. Also true that unless designed specifically, a request still could reach a container just before it's runtime does STW, causing slower UX
@@bfg5244 or they hit a perfect storm of enough of the instances start GC at the same time. "Co-ordinate GC between instances" ? "Spin up more instances"? It all sounds like a lot of "engineering" fun.
When you have 6 dependencies and each have a latency of 200-1200ms...
I’m confused what parts were bad and what parts were good
I wanted a Manicott, I compromised and ate grilled cheese from a radiator...
27:02 euh.. no, these should be parallel calls.
estimations are easy, think about it and then tell double the number ;-)
Knowing how accurate he is, is causing me stress. lol
kernel developers know why message passing kernels failed.
I'm not a kernel developer, but there are queues literally everywhere: th-cam.com/video/ezf6cV4i9Ho/w-d-xo.html
Except in many business applications, which is often not a good thing.
Bit of an odd statement, XNU/Darwin is a very solid kernel and is built entirely around the idea of passing messages.
30: 17 "People don't policy". I don't get it. Who said that I write code for people? I write services for other services to consume. And those services wanna
- authenticate themselves against other parts of the system
- store isolated user information
- talk with other 3rd party systems
- etc...
so yeah I'm gonna end up with services like facade service or profile service, etc..
How we think about things is really important because it frames how we approach things. Ideally we want to design systems around behaviours, rather than things because the goal of systems is to do things rather than represent things - representing things is often a necessity - but often we make the mistake of thinking that representing things is the end goal - it’s not. So what does that mean? Well let’s look at your Profile Service - what’s the purpose of the profile? Is it to store a customer address for ordering products? Maybe that could go in the Ordering Service? Is it to store their payment details? How about putting that in the Paying Service?
“But the customer only has one name so where do I store it?” Do you really have a customer? Someone might argue that you have an Authenticating User, a Delivery Address and a Payer. The roles of Authenticating User and Payer might be played by the same physical human, and the Delivery Address might be the home of that physical human, but logically they can be completely different. Designing your system in this way makes it far easier to cope with a change where the Payer isn’t the same human that Authenticates - an imaginary example might be that the customer is sad and you need to replace an item, so your back office staff (Authenticating User) might want to place a new order on the customers Ordering Account, where the Payer is the shop itself. I hope you can see why this conceptualisation (even if not entirely aligned to your particular domain) has benefits over noun-y Customer or Profile-like Services
Not certain what was sarcasm and what was genuine advice. The many strawman arguments made me lol, but was the only real take away "distribute dlls and don't call services"? Not sure thats great advice or particularly scalable. Maybe he was being sarcastic again.
Yes, the many layers of ironic humor makes this a mostly incomprehensible video. And he's contradicting himself several times.
@@FudgeYeahLinusLAN Its not. Heres a piece of advice from a very smart man: "There are no solutions, only trade-offs." It is perfectly possible for both things to be broken, at the same time, for different reasons. This is just a talk about "If you do this thing, expect problems."
Sometimes neither A or notA is a viable solution.
Absolutely, MicroServices cause more problems than they solve. Methodically breaking your code and data into arbitrary http daemons solves nothing but creates more problems and longer dev tool chains. The best practice of engineering is to make problems simpler and easier to solve, not expodentially harder.
Our company was recently hired to build a new system.
We're building a monolith. They ask us why no microservices, "modularization", etc..
I say: why make it more complex than needed? You're going to have 10 users. 1000 users for your internal tool is unthinkable. Our system will support thousands with no trouble.
The approach looks like GraphQL Federation.
See how easy it is to steer the system towards the disaster. I've worked on a search system that trumps the ugliness of this presenters example. I've also worked on a 'modern' system littered with excessive http fronts and event overheads. Paying too much for ASB bills. DLQ handling is treated like an extra marital child. Half of domain services are not recognizable by the product team but sitting there causing perf/financial penalties. Heck, I can give this presentation with my own experiences. Moral of this talk I think is to keep ourselves at our sharpest alerts.
I think I work at the same company as you... xD
I am not sure why you all consider http requests as blocking: I have used https requests as an async I/O using a listener to handle the response, similar to what Node.js does. You can serve other requests in the meantime. Otherwise I do agree with the overhead incurred being an issue.
He is just cheery picking bad examples to prove his point. This is similar to his queue example where he did not mention dead letter queues or putting the erroring message at the back, for example. Instead, he chose a queue where a failing message blocks all other messages, and so hence queues are bad
0:53 relatable
This is why companies should just buy an off the shelf solution that already works. Ok... if an off the shelf solution doesn't exist then programmers should create their own company and create and market such solutions. More fun than working for a dead end employer.
bad system: starting with microservices. A monolith will do the job for 99% of the apps
Amen brother.
I've never had problems with monoliths. Even with hundreds of thousands of users.
(Maybe all you guys are doing millions or tens of millions of users.. but that's not my case.. also my monoliths you can just deploy them horizontally and take on higher load)
I also prefer monoliths because nothing says job security as maintaining a code base that multiple teams work on it, new devs just hack it, tests contain static mocking & mocking & homebaked solutions, updating frameworks just take ages, we can't update DB connection manager no one wants to take responsibility, you can play security vulnerability bingo, can solve issues by just throwing more CPU and RAM on it, deployment of a new version is more eloquent than a masonic ritual, you need to become Dev Lvl 33 to know the meaning of all 267 tables in the database.
I'm only half joking, even if the talk has nice points, every solution comes with its own can of worms
After reading this video title, I thought about evolutionary reuse of a widely configurable monolith to create some kind of pseudo "macro - services" and marketing them early as new service system.
As you need >=3 times of (similar) project-expierience to improve write from scratch kills significantly.
Real big important systems evolve carefully this way, until enough know how for a better chance of new start from scratch can be risked. Otherwise a new start from scratch shows "missing hidden features" risk, nobody remembers, but many use cases rely on (undocumented), and as a theoretically nice architecture artwork (prototype, alpha, beta). If its too hard to use practically, it has low chance to replace older working system solutions and risk to be called uncompletly and not ready for daily use.
And of course, a "rewritten from scratch V2.0" that has some improvements but also some degradation of certain features of v1 will result in resistance to migration/adoption of the new version. So the engineers find themselves having to support BOTH versions in some capacity for years longer than expected. And then in order to actually reach feature parity, the shiny new v2.0 ends up with frankensteined hacks bolted on that ruin whatever elegance, simplicity, or performance improvements that were the main selling points of v2 to begin with.
@@rdean150 Yes and it creates more jobs!
Too many strawman arguments. Everything he's said up to 13:00 is either amusing, wrong, or both... I don't have high hopes for the rest of the talk.
Exactly
In the end, his ideas are even worse than the worst patterns. Coupling on code / library level instead of events... but if it works for you, fine. In the end, Microservices, Monoliths, Libraries... They all work and all suck, just in different ways.
His search engine example would fail the instant someone wants to search and get results in order of price, right?
@@TheTeknikFrik Yes any combination would still require multiple queries, just inside one central search engine (which is a plus, of course). But it does nothing to solve that issue compared to when you did not have the search engine.
Plus the Search service would have direct access to another services DB, so the deployment of that classes in there has to be coupled to the deployment of the service itself - hard without interruption, and you are now coupled on a code level instead of on the API. Congratulations. Now that may be fine because it improves other things such as performance or simplicity, everything is a tradeoff, but it is definitly not a solution to the coupling issue.
Plus you can not take advantage of the indexing featues of a product such as Elastic (with all its deficits, of course, but it still offers a lot in that regard a regular SQL DB can not).
They all suck.. But you can reduce suck by being simple. Do what is simplest. It'll be better 99% of the time. At it will definitely cost less and take less time to develop.
@@acasualviewer5861 yes that is a good idea. Stay simple as long as you have a path for evolution and fulfilling the most important qualities. But that said, what actually is simple will depend.
@@arnoldhau1 and most of the time that "evolution" never comes.. overengineering is the root of all evil.
So, how would you do microservices WELL? Do the exact opposite of the points that were listed here? Microservices are kind of nebulous to me, I've only worked with a monolith so far and it does suck, but then I see devs complain about microservices all over the internet... is there truly no good way to write code? I just really don't see what the takeaway is here.
Yeah, having the exact same problem with this talk. It's good to know that something is wrong, but I don't learn what is good from this talk.
For me personally microservices are simply a different way to organize your development workflow. They reflect the way you develop things and not how the 'domain' is structured. Product service makes sense if you have team dedicated to it which is constantly 'messing' about with it while presenting a stable interface for other teams which need to consume the service.
If you have one team then there is (usually) no need for a microservice architecture.
Of course you will have functionalities which 'touch' multiple services ... its just the way of the world. You just have to figure out how to organize your workflow to take this into account.
Well.. what if there is no way of doing microservices well? Surely if microservices are so fantastic, in 2023 there should be plenty of examples of how to build them, and we wouldn't be limited to strange videos with ironic humor explaining what to do by only saying what not to do. Right?
Rule of thumb: you can't write good microservices system if you can't write good monolith.
@@FudgeYeahLinusLANhere is plenty. It's that microservices are not for startups, they are for big companies. I work at ecom project. We have 7k microservices and 4k engineers, all working fine. It's achievement of our platform engineers, and our company is willing to pay them, although they are not profiting them directly.
Some of their work results are:
- Custom s2s, s2i for safety and control of service interactions
- Only protobuf in rpc and Kafka.
- Protofile vendoring
- No cloud, we have our own PaaS
The guy is funny as hell and I enjoyed watching it for this reason. He has a talent of a standup comedian. However his comedy style is that of a very nihilistic pessimistic cynical etc so in the end a bit too bitter.
On the negative side, he is really bad at making arguments. The only argument he gives why big bang rewrite won't work is his own experience that they just won't. "Trust me i'm a developer and we're really bad at giving estimates". The rest of the story is just his single negative experience and that is certainly not the only option. Same thing at Search engines - he constantly calls the software design "your cool stuff" as to make fun of it and consider it detached from reality by default. This is another arbitrary assumption and he doesn't go into the matter of a thing, just tries to be funny and I think the audience showed a little disappointment too (also read the comments to the video). This is just a funny rant, not a serious software discussion, sorry guys.