This was sooo good and accurate. Literally dealing with the same problems they "mocked" with overengineered microservices that break altogether when one is not available even if the point of microservices is achieving the opposite. 50 minutes well spent🙂
The way Hadi asks "Why? Why do I need to understand all of this?" This is a brutal morally degrading question I have had to ask in the past looking at the bloated legacy monoliths and wondering how can programmers make so much of mess? Do they not care at all. I felt it. 3rd time en-counting.
It’s not healthy (neither for the “programmers”, nor for yourself) to judge programmers that came before you. It’s more helpful to be respectful to what these programmers knew what they knew at the time and what other constraints they experienced back then. ;)
Nobody builds a bloated legacy monolith You always start with a simple, elegant system that solves the problem in a straightforward way Then new requirements pop up that get added here and there, and you think "oh, no problem, I just attach it to my system on this corner here where it shouldn't bother any of the core functionality" Eventually new people join the team and add things in a way that isn't what the original authors intended Soon enough, a few years and hands later it becomes a bloated confusing mess that nobody can understand And then it gets completely rewritten into a new simple elegant system over a couple of years by a team ten times the size of the original And a few years pass and what do you know... It's "the legacy system" for the next generation Stick around long enough and you get to see the cycle repeat multiple times If you're attentive enough, you realize you're writing legacy code right now Because legacy code is just code somebody else wrote, and you realize the you of next year is not the same you of today
@@neko6 I disagree. Building legacy is a style of its own. I've seen people build code real-time that's legacy the moment their hands leave the keyboard. You need to spend the effort to make it maintainable, but you can choose not to. If you don't, well, your code is going to be a hell to the next person coming in to it. It might even be you after lunch break.
Great discussion. I am glad somebody have the same feelings. Sometimes I feel like I am the only downer :D I've been working on a program for a while now where every screen, every form, every workflow is customizable but the app is used practically by just one customer that does't need it. That amount of code to support this and the amount of testing necessary to keep this going is just ridiculous. And of course. They want microservices as well.
Interesting discussion. As a relatively inexperienced Java engineer from an academic background, I often feel pressure to make my code "more corporate", that more complexity is somehow "more professional". It's reassuring to hear that manufactured complexity is the problem, not the solution. It's about finding the right balance for your domain based on the information and use cases available to you at the time of writing.
I will rephrase this a little by saying that each project should have it's own philosophy, there is no silver bullet for the specific domain solution of the project that is out-of-box. The important thing is to communicate this idea to other memebers of the team, so they can have the same level of clarity and intent. I had and still have arguments with people that apply blindly design patterns and encourage others to do the same, they have their place, however a lot of times not using them, or using them in a partial way, is more beneficial to the project. A great place to see this applied in practice very well are OSS projects that are successful and have a lot of active contributors that come from different backgrounds. Why this is more effective in OSS than corporate projects is that the future of the project literally depends on how well the complexity is managed, opposite to a corporate setting where a lot of times the rules are dictated by management and money. They are also a great place to see different philosophies at play (especially if you take a look at projects that use FP or something else than java OOP).
"“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.” -- Edsger Dijkstra almost every large system these days is unnecessary complex. Some prime examples are: - Kubernetes - Java - Object Oriented programming in general The cost of technical debt and fragility introduced as a result are highly underestimated
This is exactly the problem of the industry right now! Every sentence of the talk is correct. There is to much overengineering because those FAANG companies solved problems most of the other businesses don‘t have. But the employees of those other companies focus more on fancy tools than on the business problem they really have themselves!
Besides creativity and problem solving, I think it's also perfectionism that leads developers to over-engineer. Maybe also, developers who want to advance in their careers (with the best of intentions) try to show that they have passed the "beginner" level, and that they no longer write "naive" code...
I think we bore easily. Engineers will increase the complexity of a solution until they feel that they can just cope, but not take into account lagging indicators, and so overshoot.
"I was creating these abstractions and I was creating interfaces that only ever had a single implementation." Even a single implementation should be referenced by the interface instead of the concrete class. How do you mock for unit test then if you don't use interfaces? Do you test by using the actual implementation and dependencies? I've worked in a company where the code mostly has concrete classes as dependencies instead of interfaces. It was a pain to make unit tests.
Then you have a reason to create, abstract out, that interface; the ability to properly unit-test other classes/components that is depending on it. But don't create an interface for a class that is part of your context/platform/framework and already readily available for use in your unit-tests (eg Strings, or Exceptions, Number, etc)
After some trial error and playing by fire, I realized that those talks that implement massive complex architecture, sometimes they ignore a lot of context or details that if you knew the context, you probably wouldn't think to copy paste the idea. A lot of us very rarely bottlenecked by scalability. Performance yes, scalability I don't think so unless you work on some giant companies and the scope will be very very very limited. recently I prefer talks or discussions that dive deep on how to untangle the complexity (internal tools, business / product with existing monolith tech, decomposing monolith). My brain power can still process that and it is a lot more applicable rather than understanding how kubernetes works wihch is very taxing for me personally..
on the micro level, a lot of my job these days is converting 20-50 lines of ratsnest to 5-10 lines, chopping out mallocs and memcpys along the way. people sure like to overcomplicate stuff.
I wish someone who was a better historian than I would study more software development history. I *remember* a lot of it, but to systematize it and such would be very interesting. For example: I said 10-15 years ago that the Javascript and web browser anarchy was like the 1980s microcomputer profusion but I think that turns out to have been wrong.
I find lately the opposite to what Kelvin and Hari mention about over engineering. Lots of code with little to no design, big classes filled with logic and large data classes that are passed around (and developers not being aware of the idea of objects having data and behaviour). In talks with developers it's often mentioned that they don't design because they might not need the flexibility a good design provides, but with that they throw away what Kevlin mentioned at the start: building software is an exercise in managing complexity. The main source of over engineering I've seen lately comes from consultancies offering solutions that are silly complex (e.g. Deloitte using Mule in most engagements even if the client needs a couple of applications that integrate 3rd parties).
Although it is hard to understand and it will take enormous effort to simplify, be DISCIPLINED and try to do it - Is this what they are trying to say? :)
I disagree with the pessimism about React: it does make things like updating the DOM depending on application state much simpler. Doesn't mean it's always the right tool for the job though. The biggest problem is that best practices and idiomatic use of React has changed a few times over the years.
Unlike what says its title (managing complexity), this talk does not really help identifying, understanding and avoiding complexity nor over-engineering... Instead, it just caricature complexity, and assimilate it too often to over-engineering while trying to get its mysterious causes out of this messy analysis... For example, some "over-engineering" patterns that are discussed are simply "principles" (like dependency injection)... And by definition, a principle is a fundamental rule whose appliance is unconditional... As OOP developers, we have been taught to apply these principles to make the code-base and the team react better to "future changes" in requirements, which is supposed to lead to a "maintainable" system. But obviously, nobody can possibly predict 100% for sure what, when, and if, things will change. If it does, then the design will not be over-engineered, even though it was before the eventual change happens... And so, it is correct to say that unless designed by a time traveler (like these experts judging retroactively code), a "maintainable" system will by definition always be "over-engineered" (more or less)... We have to assume that. Maintainability (which is addressed by OOP paradigm, SOLID principles, design patterns, etc), like any other quality attribute of a computing system (scalability, availability, etc), unavoidably induce complexity in this system. Over-engineering is designing the system for features or such quality attributes stakeholders don't need (yet). We may very well find such talks in each era of software engineering. OOP by itself is over-engineering if you don't need a maintainable system. And we can keep falling back to assembler thinking that way, and then realize that yep, modern software engineering is complex, and will remain, but for different and more acceptable reason. Because notice this low to high level evolution from assembler to OOP languages... Each step of this evolution tackle lower level conplexity by abstraction... The remaining growing higher level complexity is not due to the solution but to the problem it's solving... Real world needs become more and more complex, and so solutions also. This is an acceptable reason of complexity, and one that must be distinguished.
While there’s some truth in that they’re going to one extreme caricuratizing design decisions that aren’t justified, saying that design decisions are made purely for “maintainability” is just as reductionist. I prefer the term “fit-for-purpose”, which is what they were hinting at when they spoke about how code seems to rarely be engineered with context in mind. Maintainability is one aspect of such a context imho. The knowledge and skills of the team would be another one.
@@TimSchraepen If not exclusively for maintainability of a codebase, I would be very interested if you could help me identifying what are the other purposes of OOP paradigm, SOLID principles, and design patterns! What are their other possible benefits? All benefits I can imagine of at this time fit exclusively in "maintainability" quality attribute which you might have reduced?
@@clementcazaud8040 hmmm, good question! I guess there’s the danger of going in to a semantics discussion, because one could argue that “extensibility” of a system also belongs to “maintainability”. Same goes for “readability”, which would be pointing more specifically to the assumption of the code readers (aka the team members’ skills and knowledge). OOP/SOLID patterns are there for such purpose of extensibility, but not necessarily for readability (the context there is: if you know the pattern it will be helpful, if you don’t, it might not). If you’re familiar with the 4 rules of simple design, it’s what’s meant with the 4th “rule”: “least amount of elements”. It’s an attempt at balancing out the previous “rules”, especially #2: “reveals intent”. If you think extensibility, readability and maintainability are all the same, I no longer see the point in arguing, and I’d be ok with that. I hope however that I was able to convey the distinction between the different aspects of maintainability.
They should have talked about the Agile process that force SE to only think about the short term and start the implementation right away with their adore tools without thinking of every aspect or use cases.
Focusing on accidental complexity due to over-engineering makes no sense, 90% of the time it comes from under-engineering that fails to cope with the essential complexity.
Good idea to talk about complexity. Maybe don't spend half the time whining about modern languages though. We need simple tools to help build increasingly complex client demands. The code should be simple. Complex apps can be built simply. That doesn't transfer to mean everything should be a machine code written CLI.
Ok, these guys sound like angry old men lol… I think they miss the point that the tools they bag were built because there were big flaws in the processes they replace… but they make a good point about these tools being crazy complex and not properly abstracted… like the Kube files or azure pipeline files are insanely complex and verbose
This was sooo good and accurate. Literally dealing with the same problems they "mocked" with overengineered microservices that break altogether when one is not available even if the point of microservices is achieving the opposite.
50 minutes well spent🙂
50 mins earned.
The way Hadi asks "Why? Why do I need to understand all of this?" This is a brutal morally degrading question I have had to ask in the past looking at the bloated legacy monoliths and wondering how can programmers make so much of mess? Do they not care at all. I felt it. 3rd time en-counting.
It’s not healthy (neither for the “programmers”, nor for yourself) to judge programmers that came before you.
It’s more helpful to be respectful to what these programmers knew what they knew at the time and what other constraints they experienced back then. ;)
Nobody builds a bloated legacy monolith
You always start with a simple, elegant system that solves the problem in a straightforward way
Then new requirements pop up that get added here and there, and you think "oh, no problem, I just attach it to my system on this corner here where it shouldn't bother any of the core functionality"
Eventually new people join the team and add things in a way that isn't what the original authors intended
Soon enough, a few years and hands later it becomes a bloated confusing mess that nobody can understand
And then it gets completely rewritten into a new simple elegant system over a couple of years by a team ten times the size of the original
And a few years pass and what do you know... It's "the legacy system" for the next generation
Stick around long enough and you get to see the cycle repeat multiple times
If you're attentive enough, you realize you're writing legacy code right now
Because legacy code is just code somebody else wrote, and you realize the you of next year is not the same you of today
@@neko6 I disagree. Building legacy is a style of its own. I've seen people build code real-time that's legacy the moment their hands leave the keyboard. You need to spend the effort to make it maintainable, but you can choose not to. If you don't, well, your code is going to be a hell to the next person coming in to it. It might even be you after lunch break.
Great discussion. I am glad somebody have the same feelings. Sometimes I feel like I am the only downer :D
I've been working on a program for a while now where every screen, every form, every workflow is customizable but the app is used practically by just one customer that does't need it. That amount of code to support this and the amount of testing necessary to keep this going is just ridiculous. And of course. They want microservices as well.
Interesting discussion. As a relatively inexperienced Java engineer from an academic background, I often feel pressure to make my code "more corporate", that more complexity is somehow "more professional". It's reassuring to hear that manufactured complexity is the problem, not the solution. It's about finding the right balance for your domain based on the information and use cases available to you at the time of writing.
I will rephrase this a little by saying that each project should have it's own philosophy, there is no silver bullet for the specific domain solution of the project that is out-of-box. The important thing is to communicate this idea to other memebers of the team, so they can have the same level of clarity and intent. I had and still have arguments with people that apply blindly design patterns and encourage others to do the same, they have their place, however a lot of times not using them, or using them in a partial way, is more beneficial to the project.
A great place to see this applied in practice very well are OSS projects that are successful and have a lot of active contributors that come from different backgrounds. Why this is more effective in OSS than corporate projects is that the future of the project literally depends on how well the complexity is managed, opposite to a corporate setting where a lot of times the rules are dictated by management and money. They are also a great place to see different philosophies at play (especially if you take a look at projects that use FP or something else than java OOP).
Great format! Thanks for your insights both.
"“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.”
-- Edsger Dijkstra
almost every large system these days is unnecessary complex. Some prime examples are:
- Kubernetes
- Java
- Object Oriented programming in general
The cost of technical debt and fragility introduced as a result are highly underestimated
This is exactly the problem of the industry right now! Every sentence of the talk is correct. There is to much overengineering because those FAANG companies solved problems most of the other businesses don‘t have. But the employees of those other companies focus more on fancy tools than on the business problem they really have themselves!
Besides creativity and problem solving, I think it's also perfectionism that leads developers to over-engineer. Maybe also, developers who want to advance in their careers (with the best of intentions) try to show that they have passed the "beginner" level, and that they no longer write "naive" code...
I think we bore easily. Engineers will increase the complexity of a solution until they feel that they can just cope, but not take into account lagging indicators, and so overshoot.
Two masters on the seat. Loved hearing the bitter truth sprinting. Wish I could gain hair like Henney's.
"I was creating these abstractions and I was creating interfaces that only ever had a single implementation." Even a single implementation should be referenced by the interface instead of the concrete class. How do you mock for unit test then if you don't use interfaces? Do you test by using the actual implementation and dependencies?
I've worked in a company where the code mostly has concrete classes as dependencies instead of interfaces. It was a pain to make unit tests.
Then you have a reason to create, abstract out, that interface; the ability to properly unit-test other classes/components that is depending on it.
But don't create an interface for a class that is part of your context/platform/framework and already readily available for use in your unit-tests (eg Strings, or Exceptions, Number, etc)
After some trial error and playing by fire, I realized that those talks that implement massive complex architecture, sometimes they ignore a lot of context or details that if you knew the context, you probably wouldn't think to copy paste the idea. A lot of us very rarely bottlenecked by scalability. Performance yes, scalability I don't think so unless you work on some giant companies and the scope will be very very very limited.
recently I prefer talks or discussions that dive deep on how to untangle the complexity (internal tools, business / product with existing monolith tech, decomposing monolith). My brain power can still process that and it is a lot more applicable rather than understanding how kubernetes works wihch is very taxing for me personally..
on the micro level, a lot of my job these days is converting 20-50 lines of ratsnest to 5-10 lines, chopping out mallocs and memcpys along the way. people sure like to overcomplicate stuff.
I wish someone who was a better historian than I would study more software development history. I *remember* a lot of it, but to systematize it and such would be very interesting. For example: I said 10-15 years ago that the Javascript and web browser anarchy was like the 1980s microcomputer profusion but I think that turns out to have been wrong.
I'm not sure what you mean, but if you're likening the browser wars to proprietary early 1980s architectures, I'd say that may be a fair comparison
I find lately the opposite to what Kelvin and Hari mention about over engineering. Lots of code with little to no design, big classes filled with logic and large data classes that are passed around (and developers not being aware of the idea of objects having data and behaviour). In talks with developers it's often mentioned that they don't design because they might not need the flexibility a good design provides, but with that they throw away what Kevlin mentioned at the start: building software is an exercise in managing complexity. The main source of over engineering I've seen lately comes from consultancies offering solutions that are silly complex (e.g. Deloitte using Mule in most engagements even if the client needs a couple of applications that integrate 3rd parties).
Although it is hard to understand and it will take enormous effort to simplify, be DISCIPLINED and try to do it - Is this what they are trying to say? :)
"I wrote the other day on this social media, Its called Twitter" 😄
Nice talk...
I disagree with the pessimism about React: it does make things like updating the DOM depending on application state much simpler. Doesn't mean it's always the right tool for the job though. The biggest problem is that best practices and idiomatic use of React has changed a few times over the years.
why spent compute on solving the problem if you can spent compute time calling rest apis
Part of the problem is, of course, developer advocates :)
Hahaha, i know right
Unlike what says its title (managing complexity), this talk does not really help identifying, understanding and avoiding complexity nor over-engineering... Instead, it just caricature complexity, and assimilate it too often to over-engineering while trying to get its mysterious causes out of this messy analysis...
For example, some "over-engineering" patterns that are discussed are simply "principles" (like dependency injection)... And by definition, a principle is a fundamental rule whose appliance is unconditional... As OOP developers, we have been taught to apply these principles to make the code-base and the team react better to "future changes" in requirements, which is supposed to lead to a "maintainable" system. But obviously, nobody can possibly predict 100% for sure what, when, and if, things will change. If it does, then the design will not be over-engineered, even though it was before the eventual change happens...
And so, it is correct to say that unless designed by a time traveler (like these experts judging retroactively code), a "maintainable" system will by definition always be "over-engineered" (more or less)... We have to assume that. Maintainability (which is addressed by OOP paradigm, SOLID principles, design patterns, etc), like any other quality attribute of a computing system (scalability, availability, etc), unavoidably induce complexity in this system. Over-engineering is designing the system for features or such quality attributes stakeholders don't need (yet).
We may very well find such talks in each era of software engineering. OOP by itself is over-engineering if you don't need a maintainable system. And we can keep falling back to assembler thinking that way, and then realize that yep, modern software engineering is complex, and will remain, but for different and more acceptable reason. Because notice this low to high level evolution from assembler to OOP languages... Each step of this evolution tackle lower level conplexity by abstraction... The remaining growing higher level complexity is not due to the solution but to the problem it's solving... Real world needs become more and more complex, and so solutions also. This is an acceptable reason of complexity, and one that must be distinguished.
While there’s some truth in that they’re going to one extreme caricuratizing design decisions that aren’t justified, saying that design decisions are made purely for “maintainability” is just as reductionist.
I prefer the term “fit-for-purpose”, which is what they were hinting at when they spoke about how code seems to rarely be engineered with context in mind.
Maintainability is one aspect of such a context imho. The knowledge and skills of the team would be another one.
@@TimSchraepen If not exclusively for maintainability of a codebase, I would be very interested if you could help me identifying what are the other purposes of OOP paradigm, SOLID principles, and design patterns! What are their other possible benefits? All benefits I can imagine of at this time fit exclusively in "maintainability" quality attribute which you might have reduced?
@@clementcazaud8040 hmmm, good question!
I guess there’s the danger of going in to a semantics discussion, because one could argue that “extensibility” of a system also belongs to “maintainability”. Same goes for “readability”, which would be pointing more specifically to the assumption of the code readers (aka the team members’ skills and knowledge).
OOP/SOLID patterns are there for such purpose of extensibility, but not necessarily for readability (the context there is: if you know the pattern it will be helpful, if you don’t, it might not).
If you’re familiar with the 4 rules of simple design, it’s what’s meant with the 4th “rule”: “least amount of elements”. It’s an attempt at balancing out the previous “rules”, especially #2: “reveals intent”.
If you think extensibility, readability and maintainability are all the same, I no longer see the point in arguing, and I’d be ok with that. I hope however that I was able to convey the distinction between the different aspects of maintainability.
They should have talked about the Agile process that force SE to only think about the short term and start the implementation right away with their adore tools without thinking of every aspect or use cases.
Folder for decorators: guilty 🙋♂️
The first step is admitting it
Focusing on accidental complexity due to over-engineering makes no sense, 90% of the time it comes from under-engineering that fails to cope with the essential complexity.
Back to basic.... KISS!!!! 🤣🤣🤣
Good idea to talk about complexity. Maybe don't spend half the time whining about modern languages though. We need simple tools to help build increasingly complex client demands. The code should be simple. Complex apps can be built simply. That doesn't transfer to mean everything should be a machine code written CLI.
Ok, these guys sound like angry old men lol… I think they miss the point that the tools they bag were built because there were big flaws in the processes they replace… but they make a good point about these tools being crazy complex and not properly abstracted… like the Kube files or azure pipeline files are insanely complex and verbose