The PROBLEM With DORA Metrics
ฝัง
- เผยแพร่เมื่อ 10 ก.พ. 2024
- Are DORA metrics causing more problems than they solve? Allen Holub explains why he thinks that accelerate / DORA metrics don't really cover developer productivity like lots of organisations believe.
This is a clip taken from Allen's FULL Engineering Room episode, which you can watch HERE ➡️ • Agile & Scrum Don't Wo...
___________________________________________
🙏The Engineering Room series is SPONSORED BY EQUAL EXPERTS
Equal Experts is a product software development consultancy with a network of over 1,000 experienced technology consultants globally. They increase the pace of innovation by using modern software engineering practices that embrace Continuous Delivery, Security, and Operability from the outset ➡️ bit.ly/3ASy8n0
___________________________________________
#agile #softwareengineer - วิทยาศาสตร์และเทคโนโลยี
Goodhart's law: "when a measure becomes a target, it ceases to be a good measure"
this is what the clip is all about
"Tell me how you measure me, I will tell you how I behave.", Eli Goldratt
My law: Target your metrics or your metrics will target you.
Software engineering is knowledge work. Now let's compare software engineers to other knowledge workers... Do we measure lawyers by the number of legal documents they serve? Do we measure doctors by the number of prescriptions they write? Do we measure (structural) architects by the number of buildings they design? Of-course not! What matters is the contents of the legal document, which medicine is prescribed, the functionality and visual aesthetics of a building. As it pertains to software engineering, what matters is the relevance of what the code does as well as the maintainability and releasability of said code. Sometimes the best thing a software engineer can do is reduce the amount of code required to get something done.
In many large US healthcare systems, doctors are measured by the number of patients they see per day and the amount of time they spend with each patient. I don’t think anyone believes these measurements lead to better outcomes for patients. It’s not hard to find attempts by companies to industrialize knowledge work in nearly every domain.
@@jasondbaker When McKinsey comes to town...implying that no doctor in their right mind would construe such a system. It is bound to have serious and even life threatening consequences, as it would encourage professional negligence, not to mention exposing the healthcare providers to class actions and litigation. Trying to "industrialize" knowledge work, is something typically pursued by managers, or management consultants, with very little knowledge of the domain in which they find themselves managing, so they resort to Taylorism, staid accounting techniques and so on. Arguably this is a psychological crutch, to help manage their discomfort by restoring a sense of being in control when they are really out of their depth and don't want to admit it, not least to themselves.
Such an interesting discussion. It's awesome to see the sides of an argument. I think Dave posting counter views to his own on his channel shows how open minded he is. Personally I think the truth might be somewhere in the middle. Dave sees the value of metrics when they are understood correctly, while Allan sees them as always being used incorrectly because most people don't understand them or can't, except perhaps the engineers in the dev team, so if they do use metrics to help them improve, they should never share them with management, let managemt use their own metrics. And if they use the same metrics, challenge them, because they are not meant to be used like that. Management should focus on measuring outcomes and nothing more.
Wow. That was superb. Thanks for excellent video. Thanks Allen Holub. Great critical appraisal. Freedom to experiment is so crucial to good software development. It defines us.
Really important to understand that metrics are FEEDBACK. They can be used to decide if you need to change something. You still have to REASON about the feedback from the metrics.
We actually had the same issue exactly, management were looking for numbers, without regard to the story behind it
We are also looking to bring DORA metrics to the teams and it is very funny to see you have the same discussion. In the end we came to the same kind of conclusions. I feel kind of validated here. Sincerely thank you.
I'm curious what you think of the RICE metric for estimating value
Just indicative.
If your confidence is low, then work to increase confidence rather than ignore potentially valuable stories
If estimates are high, then work to break down the stories into smaller ones
Don't go crazy with a spreadsheet and values for this, just use it comparatively between two stories
Whilst I generally find Allen a bit extreme in his views, the point around DORA Missing a metric when it comes to value is super valid.
Not having a metric or measure on value and outcomes means that a lot of management / leadership are easily tempted into these metrics becoming productivity metrics, the key issue for me is when things turn into individual people productivity is super dangerous - team focused productivity is more helpful as you could understand and fix systemic impediments but still deliver crap and unfortunately leaving a big hole in the approach (e.g. oh that is product's job rather than the developers.. that is bad, the whole team needs to care about value for the customer).
Absolutely right in saying that developing software is not the same as running a factory.
Developing software is the same class of work as developing a new product. No-one tries to develop new product in a "lean" way. And we shouldn't try to develop software like that either. It isn't called software _development_ for nothing.
The problem is, that work expands to the available time and not every project is the fancy motivating next big thing creating tons of value. Many developers work for months on the change to a tax calculation, that must be done until the end of the year or the company is in big trouble. Developing software is not like a factory, but the DORA metrics (or better: the CI principles) measure/define the quality of our tools. You can build a great product with sh*tty tools and the best tools might fail you, but having great tools increases the chances for a great product. Measuring "productivity" will not work until an AI is capable to really dig into project dynamics. Until then, i prefer the DORA metrics over "lines of code" or "commits per week" or whatever those dysfunctional companies use to shovel their own grave.
@@vanivari359 Yes, many SW developers gold-plate their work. But the solution is team leads & seniour devs who keep people on track and teach them (by example) that "Good is Good Enough".
Deep down, gold-plating is often a sign of insecurity. Releasing early and releasing often is the best way to cut through that insecurity.
Not sure if I understand correctly, but I feel there are people trying to develop products in a lean way...? I mean, there is something called an MVP, which classifies as lean product development, no?...
@@KiddyCut "Lean" is a method for optimizing a manufacturing process, based on 5 principles (Defining Value, Value-Stream Mapping, Creating Flow, Establishing Pull, Continuous Improvement), some of which are vaguely applicable in SW development, but certainly not all, nor the most important. It focuses on reducing waste by manufacturing only that what is explicitly required, cutting stocks, shortening supply lines, etc.
If done right, it starts with holding Top Management responsible & accountable for waste in current processes. That often means publicly humiliating the CEO, if he is reluctant to take responsibility. Which is why usually it is not done properly, and has gotten a bad name as a result.
No one tries to develop new products in a lean way? That seems like an overly broad claim. I have a bookshelf with several books focused on lean product development and lean startup methodologies. Clearly some companies are following these practices when developing products.
Oof, that last comment. The head agile coach at my job insists we never move things left on the kanban board, only right. That felt icky but i couldn't articulate why. But now i see that it promotes linear thinking
After working on hundreds of projects using Kanban, I can’t think of many cases where we wanted to move something left. Maybe there are cases where we started working on a project and decided to postpone work due to new priorities. Now that project isn’t really in progress anymore so you would like to move it left to address WIP limits. The biggest issue with moving work left is that it can muck up your work metrics if you don’t account for it.
I believe in metrics. Metrics like test coverage, Cyclomatic Complexity, Hot Spots for bugs & code modifications, etc.
NOT in metrics that would "measure" the contribution of a human being to the team.
But I agree there shouldn't be hard limits set them, but they should help prioritize the work.
The DORA metrics does not measure the contribution of one member to a team, but the performance of a dev ops team,. right?
"there's something wrong figure it out", that's by design right?
My management isn't enlightened about lean, the do however defer to me (tech management) when it comes to the metrics.
Problem is that simple metrics are too easy to falsify - release even if you've made no changes! Some years back I worked at a company where we collectively had a target to remove all warnings. 0 warnings on compilation. Pretty easy to measure, surely? It emerged that one team had literally disabled warnings by the compiler flags! OK some senior people involved left the company soon after, rumour had it under a cloud, but...
Metrics could show that tehere is weak spots to working it out and see helps it or not. It is not about productivity, it's about finding ways to improve things
What I have found with the DORA metrics is that releasing frequency is something that is often really difficult with scientific software. Sometimes it takes us a month to deliver a feature because of all the papers that have to be read and then implementing the math. While we do add the feature in small pieces and unit test all of it none of it works until it is all done. You can't really use just part of a new physics model without it being complete.
DORA doesn't say the release has to contain value, just that you do it often. That way at least it's integrated often, and is in a releasable state.
So when your final commit comes through that finishes the feature and makes it work end to end, you can deliver that more quickly than if you didn't have a robust ci/cd process.
Assuming the metrics are used well and not abused, as Alan is saying haha.
@@manishm9478 We have a CI/CD system and full unit testing. I would just not consider it released until it actually works. We keep it in a constantly working state but what value would a release have for just part of a pysics model?
@@Immudzen fair :) sounds like you have an awesome software delivery process in place already!
Oftentimes developers work on features that take more than a day to build and they have the same argument: “I don’t want to release my code until the feature is complete.” Sharing code and releasing it don’t have to be tightly coupled. There are a couple patterns you can follow during software development to address this. I guess my point is that the hesitancy to share code before feature completion is not unique to scientific software. It’s a challenge that all software teams face and the solutions are likely similar.
@@jasondbaker We share the code right away and we have tests to make sure the fragments work. I would just not consider it released if it is just a model fragment. You can't simulate part of a physics model. It has no value to anyone and can't be used but the code is there and tested.
Quality metrics are DEscriptive not PREscriptive. Once was at a company that wanted 100% test coverage on new code for a codebase that was previously almost entirely untested. The result was predictable: a bunch of brittle "unit" tests that didn't even properly cover all scenarios. In general, our code was better in the areas that were unit tested, likely because the devs that cared enough to unit test were also writing code that could easily be read and tested, but instead of the whole app magically rising to meet this standard of quality, it killed the metric and gave us a bunch of random noise to deal with whenever we made literally any change in those areas. It also completely busted our build times because Moq (at the time) had a weird quirk where running 30k unit tests with any parallelism would occasionally cause test failures due to a dll conflict, and that likelihood increased with more tests.
I don't think the problem is "metrics escaping to management", but rather "management constantly demanding additional metrics".
IRL example of things going south. A chain store that won't be named started measuring cashier "productivity" as "amount of items registered per hour". Guess. Cashiers started registering items individually instead of *x. Productivity skyrocketed and so did the lines at the register. Store course corrected and demanded *x or else. Things improved, but... not as much as they expected. They found out throwing two items fast (you have two hands) is faster than *2. And people buy A LOT of stuff in pairs. So, change that to *n if n>2 (software will aggregate for sum). Better, but still sub optimal, because now the cashier had to eyeball it and make sure they didn't throw 3. So... they just scrapped the idiotic metric. Did they come up with another idiotic metric? You bet, but that's another story... ;)
I must say I disagree with Mr. Holub here on most of his points. He called out release cadence and time to recovery specifically and he is wrong on both. What DORA showed perhaps counterintuitively is that speed == quality. While you can certainly get trapped into "release crap faster" if you maintain your ability to release daily or hourly you can actually fix that quite easily. But if you ship crap weekly fixing it would be much harder. Same applies for time to recover. If you recover in minutes than any screw-up is perhaps mild inconvenience. If your time to recover is in hours or days or more than any prod incident will be utter disaster. So if you want to experiment, rollback etc. you'd better make sure you can recover quickly an deploy/rollback even faster. Which is what DORA shows. If you go the opposite way the same metrics will tell you that you will end up gradually in worse and worse spot. Recovery wil ltake longer, releases will slow down further (bi-weekly, monthly) and quality will go down with them however you measure it.
And lastly about Kanban. Where is the notion that Kanba == going forward comes from? Kanban is a way to organize work. If you prioritize tasks to rework stuff, rollback stuff etc. you will have the easiest "circular" iteration experience. Certainly better than having that opportunity every two weeks or what your sprint length is. That people misuse it into thinking that done tasks cannot be revisited is not the problem of Kanban but those people...
Full agree on the management though. Middle management is the killer of software in big companies. Where top management wants the products to do well so the company does well and the devs want to do their work well the goals of the middle management are vastly different to those two groups. Hence they so often become the main problem. Because top level management would never "weaponize" any metrics simply because the only metrics that matter to them are the sales / performance of the product and company. But middle management would happily weaponize any metric that would make them look good in the eyes of the brass.
Cant measure value? At all? I get the context specific argument but if you cant measure value, how do you know you're moving in the right direction? The product manager just says "trust me it's good."?
Money is the real measure of value what the customers pay for the software. When you are developing something big you do a market analysis to try to find out what is valuable. Till you do not get paid you can work with only predictions.
@@banatibor83 most people in the lean space say ship it and charge your customers as early as you possibly can to validate. Then look at all aspects of your business model with data - how much does it cost to gain or keep a customer vs chirn rate and what is needed to make the product actually worth it. Then find out through tuning product, marketing churn etc to see if it is viable.
The point about "you can't choose one of the dora metrics" makes me wish someone had a good formula to combine all the dora metrics into a single number that could be used as a useful measure of software delivery performance and a leading indicator of outputs.
“Everything should be made as simple as possible, but no simpler.” I get the idea of wanting the metrics even further simplified, but you'd lose what makes them useful by doing so.
@@manishm9478 Lots of people say we shouldn't aim to optimise just one of the metrics, we need to monitor all of them together. So that sounds it's already a recommendation to synthesising them in some way.
I guess one way to come up with a single number is to just have a human look at all the metrics and then answer a 5 point Likert style question: "To what extent do you agree with the statement 'These are good Dora metrics'. But that would be very subjective.
@@barneylaurance1865 i'm curious, could you explain your use case a little more? Do you want to use a synthesised number to report to other people outside the team?
@@manishm9478 I don't have a very specific use case in mind. I think it would more be just for the team to track how well it's doing on the things DORA measures for itself, and hopefully to think about and tie back to actions they've taken that have helped or hindered.
I have seen managers reporting one or two of the DORA metrics to outside the team though, so if there was a credible formula to aggregate the metrics into one number that could be a way to avoid cherry picking the metric that looks most favourable.
I agree that DORA metrics aren't perfect - and like all metrics - there is always someone in management trying to misuse them (look at lines of code for another example), but DORA is still the best we have. I think it's good to have discussions about how not to misuse these metrics, but I haven't seen anything better than DORA, that isn't really high-level hand-wavy "we need to measure outcomes not activities" sort of discussions.
To my ears, no one makes as much sense as Allen Holub. Mr Farley a close second :-)
Interesting. Sounds to me like Allen's business clients either need to back off s/w dev, or learn more about its management. The comment on how inappropriate "Lean Thinking" is may apply to dev, but it's a more useful approach in an Ops /ITSM context (e.g. measuring and managing down variability as a source of deliver failure.)
I very much like Alan’s focus on outcomes but I’m afraid he’s got Lean completely wrong. There’s nothing in Lean against experimentation. Rather the contrary: one of its tenets is “Amplify Learning’.
Btw, DORA’s MTTR is not about failed experiments, it’s about broken environments.
Agreed. He also forgets about the change failure rate metric. Smaller batch sizes of work allow us to change software more frequently and mitigate the risk of change failures. Any experiments are likely small and easy to unwind.
This isn't so much a problem with metrics... as Allen describes things the problem is "management".
I always cringe when I hear Allen and many others like him speak of management as a threat or an opponent. That kind of self-isolation of teams within an organization is the opposite of what agile is supposed to be. Communication have to flow in order for collaboration to happen. It is sad that Allen seem to have been in so many abusive situations with management people that are either assholes or psychopaths that he has come to think of all management to be this way.
I do think that in many organizations that are dysfunctional with bad management, Allen is very right that those people will weaponize everything. Any organization where profit or winning is more important than people or producing great products is an organization I avoid like the plague. Profit is important, but the way to get there varies greatly, and bad organizations will abuse any and all methodologies and sacrifice anyone to "win".
DORA metrics in such organizations will be weaponized and abused, just like Allen says. Fortunately, not all companies are such horrible places to work in. I think culture play a big role in that type of workplace you will find in different parts of the world, and I am happy that here in Sweden those kinds of bad workplaces are the exceptions and not the rule.
Personally I don't believe that value can not be measured, but we are very bad at measuring many things if they are not directly connected to money for example. While it is very easy to measure outcome, like number of sales or signups connected to our CTAs in terms of conversion rates, we can't measure smiles or frowns. People might hate what we build, but the price is good, so we put up with it. This is of course the opposite of what we want, but it still measures well in terms of value creation.
So what we need to do as an industry is to find out, how do we measure things that do not have hard values? This is also important in society, as we today can't really quantify the value a teacher or mentor might provide. How do you quantify the value of a police officer talking you down from a suicide attempt, or a firefighter preventing your house from going up in flames by risking their lives to fight a forest fire?
Same thing for us. How do we measure when a developer creates great code and help define the right solution to a problem that delights the users and bring value to the company? Visibility is one thing, but then what if the organization of toxic, so the bad data is weaponized?
It is a very difficult thing to do, but one of the most important ones for the future, I think.
I definitely speak from inexperience, but I struggle to find how an organization intensely focused on "winning" is a bad thing for everyone involved. I guess that I directly equate that with treating the team well enough to keep winning, since the other way around will end up "loosing" by prioritizing a small "win". By that logic, an organization focused in winning at all costs, would end up optimizing towards that, and end up making some of the decisions that make it a good place to work with. At least that is what I understand.
@@AngelLoredo53 In a perfect world that would be true, but in dysfunctional organization the winning part is not to win for the organization, but for yourself. So throwing others under the bus to promote yourself is acceptable in such organizations. These organizations tend to attract people from the psychological Dark Triad (narcissism, machiavellianism and psychopathy) and people that are more servant oriented with high empathy tend to feel abused in such environments.
So it is not the organization as such that strive to win, it is the individuals that foster a culture to sacrifice others to promote yourself.
@@AngelLoredo53Well one way that it is a bad thing, is, as you say, when it is treated naively to assume that the way for orgs to win is to take advantage of their employees which does seem like a fairly common anti-pattern. I don't subscribe to evil management, but I do think that management is poorly taught and often poorly practiced in all spheres of life. I don't think that people or orgs set out with the aim of doing a bad job, but often bad-jobs result from their actions anyway. I was once part of a very large team that had been working with one of the big consultancies. The project was doing poorly and we had all been working ridiculously long hours for nearly 18 months. Many of us had been working 80 hour weeks for months. One of the leading consultants, a senior partner in the firm, came into a leadership meeting one day and said "It's time to bring the hammer down on the dev teams", at which point I got up and left. At that point pushing people this hard we had lost of software, and after 18 months of work, it didn't even compile together, let alone work! It had started out working and working quite well and was destroyed by too many people and too much macho crap in the leadership team. Even so, I think that they thought that they were doing the right things, they were wrong, but I don't think that they were intentionally evil, just accidentally incompetent.
@@ContinuousDelivery Thanks for the response, I never knew things could get that bad, even more so knowing that the team were doing such intense work. It shows that there is only so much you can do by naively bruteforcing your way through a project. Reminds me of the so-called "Bioware magic".
It seems like Allen Holub considers agility and value to be important, and believes that he can improve development organizations' agility. So how does he know that he can improve agility if he can't measure it? How does he expect a team he's consulting with to deliver more value if they can't measure value?
Mr. Holub says that the problem is that people frequently misuse metrics. Granted; but is the solution then to stop having metrics. If someone goes to the doctor and they hear that their blood pressure is too high and then they do something stupid to bring the blood pressure down, should we stop measuring blood pressure? If "management" will make bad decisions based on misuse of useful metrics, then spend the time to help them understand the context. This should apply for DORA metrics as it should for metrics that measure business outcomes achieved through key results.
This is just ridiculous. That’s why it comes in pairs! You cant just keep deployment frequency high by deploying crap because your change failure rate would be high.
And yes, DORA does not measure outcomes! You use business metrics for that - like OKRs and KPIs.
But yes, the DORA metrics is not the end all be all. But it does facilitate faster experimentations.
If you only measure DORA without outcomes, then you essentially have an efficient feature factory. But if you measure outcomes (through your OKRs or KPIs or whatnot), then DORA can help you be more agile to get there.
It sounds like he already has a preconceived aversion to lean and DORA that he did not even try to validate his own assumptions.
I'm afraid this doesn't reflect very well on Allen here. I take very little but *metrics bad, management bad* from Allen's dialogue. Whereas Dave's point about interpreting data scientifically is expressed consistently in his other material and clearly again here. Dave has always clearly made the point about understanding the reason that this capability delivers value, and never emphasises just satisfying the metric. Allen's caution about management misinterpreting metrics is no doubt borne from experience but is seeming entrenched or habituated bordering on trauma. Managers *can* learn, and software development is getting towards the maturity where they can insist that they must learn.
On measuring "value":
This is completely domain-specific. A/B testing in e-commerce is widely adopted and can objectively be shown to have 'value' in the change in conversion rate. I have a suspicion that a lot of technology discourse is biased by an assumption of the business domain being e-commerce or something similar. There are business domains where failed transactions, customers losing interest and walking away etc are low-volume and high-value and therefore "experimentation"/"testing in production" is not acceptable.
Specifically on the deployment frequency metric:
This is not necessarily about how frequently you *do* deploy, it's about how frequently you *can* deploy. The purpose of the metric is to get you to assess how much overhead there is to you preparing a release. If releasing is hard, you are not a high-performing software team. I am sorry to say that I think Allen has missed the point here. Teams that don't have a need to release daily, and release at a cadence that is convenient for their users, can be high-performing. Their takeaway from the metrics should be 1) is their release low-overhead? 2) is their release reliable? It's not actually about frequency, it's about the robustness that permits frequency.
Allen's shtick is all about saying things that are technically correct in a very narrow context, but phrased in a way that's missing the point in the majority context. In that way he gets a lot of rage-clicks while also getting to act surprised when he gets shouted at on social-media.
It's fine to say "metrics can be abused in a dysfunctional organization", but then says "we should have a metric for value" as his solution? But Mr Allen riddle me this, if you go to a dysfunctional org and tell it to instrument a "value metric" that is driven by management, what's to say that that metric is actually well designed and wouldn't be abused?
his point is that there is no measure of value.