Big thanks to my family for the help in filming this. We had three cameras, a fancy mic, a camera slider, all kinds of lights, and even some bit roles :)
Min 0:30 - Deployment frequency is NOT "how long it takes to get a change to production". Deployment frequency = How often an organization successfully releases to production Min 0:37 - Change lead time is not "From the developer works on it all the way to it gets in production", that is called Cycle Time "Lead Time refers to the amount of time between when an order is placed and when it is delivered, Cycle Time refers to only the amount of time when actual work is done to complete an order"
You are right - I misspoke on change frequency. Good catch! As for change lead time vs cycle time, the definitions vary depends on who you are talking to. Cycle time, for example, could also include the moment when the ticket is first created or even first requested by a customer. Technically, in the DORA metrics context, change lead time measures them time the change is "ready" or committed to when it reaches production. However, in practice, measuring from PR merge to deploy isn't actually that useful, so people tend to prefer measuring from coding time to production.
Thanks, we were trying to find a way to make it a bit more interesting than just an information dump. I think you can see my dad and kid sometimes in the blurry background operating camera C :)
You mentioned for small teams working in Agile or big teams working on Monolith, smaller the shipping the better it is. But how small the packages should be delivered to Production? Deliver 1 story in an hour to production?? if it is so it also involves testing the code in Production. Can you provide more insights to that?
It depends on the product and situation, but as a general rule, the research backs up the correlation of higher deployment frequency with higher business success. For a self-contained backend microservice, a team may be able to go from coding to shipping within 15 minutes. We did that when working on email sending service in Atlassian. For a monolith that needs lots of soaking and external testing before production, weekly may be the best you can do. There are techniques that can help, such as feature flags and observability, but I think it is wrong to expect every service to deployed multiple times a day.
Should show this to my organisation who insist on having a separate QA function and have moved DevOps Engineers to Infrastructure because they think DevOps is purely about the infrastructure that applications reside on. They just don't get it.
Many thanks for this video! About the MTTR, when should we consider a hotfix instead of revert? Is it only about failure natural? Should we consider production as “cumulative stack of changes” and avoid revert even for a critical issue ?
The rule of thumb I use is what gives the best customer experience. If you are deploying frequently and have the ability to push out a fix quickly, then rolling forward is usually the right choice. However, if your deployment process is long and you have the ability to quickly jump back to the previous version, start there then roll the fix out asap. Something to remember with rollbacks is one way things like database migrations. Sometimes, a deployment may contain a database migration that is one direction and can't be reverted, so if that is something your service uses, rolling forward may be your only option. There are ways to structure and rollout database changes that are seamless and can even be reverted (old code on a new db), but they have a cost in complexity, both code and deployment process. That said, I'm usually willing to make that tradeoff.
The way how these 4 metrics are explained by him, the gentleman doesn't seem to have a lot of experience with the Infrastructure Operations, ITSM and ITIL, but he gives his perspective as developer. Fair enough!
Maybe Habit Teacher could be a headline that could spark some understanding and foster a conversation about what we do. You know I love your videos, but just to iterate. That amount of distance between the camera and you in these videos is the distance you should keep 😆
I like how you had to emphasise that it is "not about the metrics, but about the developers". Obviously just another way to micromanage developers and have micro level insights, so this message had to be stressed, kind off like propaganda brainwashing. Not a sincere product definitely. 100% agreed with @JonathanCrossland and his comment - it is not about developers, but about desire to do more quicker, which in a long term is a recipe for disaster.
There are so many bad assumptions here. These metrics do not show anything to developers. Every time you sample these metrics the context is different. Different work, task, code, requirement etc. Shipping more and smaller is wrong. Ship well and when necessary is more correct. Small changes can easily affect a lot of things. The assumption here is that a small change is small. Sometimes a small change is huge. It also has nothing to do with quicker. Being fast is nonsense for developers, it is a business desire. We should stop trying to apply business desires as a positive to developers. Change failure rate is bad, because a failure ALWAYS changes.the nature of a failure is always different, in how you measure and how it works.
I think both things can be true. The State of DevOps research showed by surveying 30+k companies that deploying frequently did strongly correlate to organizations achieving their business goals, but as you correctly point out, at the micro it is a lot more complex. I noticed you have a TH-cam dev channel, would you be up for a collab/debate/discussion/whatever? I'd love to chat further :)
Context matters here. The goal is to be capable of releasing software on demand. This typically means at the tempo of the business value proposition(features and bug fixes). The DORA metrics align to this concept, just balance them against the deployment cadence needed to achieve business value.
@@donbrown573 This is basic correlation/causation cargo cult fallacy. Just because successful people disproportionately own BMWs does not mean owning a BMW will make you successful.
DORA Metrics are metrics, can be misunderstood and misused, leadership as usual look at numbers, but the numbers doesn’t explain the story or the research and all the dependencies that caused these numbers.. and sooner or later, these metrics will be used to measure performance, and payout bonuses, it’s going to be a mess.. just my opinion
so what do you do with the metrics? "Hey Bob, start programming faster!" lol just another system to make software development more costly and convulated. Great for job security, managers hate this. Just makes devs want EASIER tasks, so they ship "faster". While poor junior get guys huge complex feature that takes forever. So those metrics tell me nothing. Just give me css changes to make those metrics awesome.
We see DORA metrics most useful at a team or service level, and not at an individual level. A team, looking to improve their performance, may see a long review lag time and decide to change their process on PR reviews. A team seeing a high change failure rate may decide to increase deployments or invest in better CI testing. I agree trying to apply these metrics to a single developer, or worse, use them to compare devs with no other context is very harmful.
Big thanks to my family for the help in filming this. We had three cameras, a fancy mic, a camera slider, all kinds of lights, and even some bit roles :)
Min 0:30 - Deployment frequency is NOT "how long it takes to get a change to production".
Deployment frequency = How often an organization successfully releases to production
Min 0:37 - Change lead time is not "From the developer works on it all the way to it gets in production", that is called Cycle Time
"Lead Time refers to the amount of time between when an order is placed and when it is delivered, Cycle Time refers to only the amount of time when actual work is done to complete an order"
You are right - I misspoke on change frequency. Good catch!
As for change lead time vs cycle time, the definitions vary depends on who you are talking to. Cycle time, for example, could also include the moment when the ticket is first created or even first requested by a customer.
Technically, in the DORA metrics context, change lead time measures them time the change is "ready" or committed to when it reaches production. However, in practice, measuring from PR merge to deploy isn't actually that useful, so people tend to prefer measuring from coding time to production.
Loved the over-the-shoulder insights. 😄 Great video.
Thanks, we were trying to find a way to make it a bit more interesting than just an information dump. I think you can see my dad and kid sometimes in the blurry background operating camera C :)
I've watched this video multiple times in the past months, very insightful, thanks!
Thanks a lot for this video. I really appreciate it.
You mentioned for small teams working in Agile or big teams working on Monolith, smaller the shipping the better it is. But how small the packages should be delivered to Production? Deliver 1 story in an hour to production?? if it is so it also involves testing the code in Production. Can you provide more insights to that?
It depends on the product and situation, but as a general rule, the research backs up the correlation of higher deployment frequency with higher business success.
For a self-contained backend microservice, a team may be able to go from coding to shipping within 15 minutes. We did that when working on email sending service in Atlassian. For a monolith that needs lots of soaking and external testing before production, weekly may be the best you can do. There are techniques that can help, such as feature flags and observability, but I think it is wrong to expect every service to deployed multiple times a day.
Should show this to my organisation who insist on having a separate QA function and have moved DevOps Engineers to Infrastructure because they think DevOps is purely about the infrastructure that applications reside on. They just don't get it.
Many thanks for this video!
About the MTTR, when should we consider a hotfix instead of revert? Is it only about failure natural?
Should we consider production as “cumulative stack of changes” and avoid revert even for a critical issue ?
The rule of thumb I use is what gives the best customer experience. If you are deploying frequently and have the ability to push out a fix quickly, then rolling forward is usually the right choice. However, if your deployment process is long and you have the ability to quickly jump back to the previous version, start there then roll the fix out asap.
Something to remember with rollbacks is one way things like database migrations. Sometimes, a deployment may contain a database migration that is one direction and can't be reverted, so if that is something your service uses, rolling forward may be your only option. There are ways to structure and rollout database changes that are seamless and can even be reverted (old code on a new db), but they have a cost in complexity, both code and deployment process. That said, I'm usually willing to make that tradeoff.
Very well explained! thank you!
The way how these 4 metrics are explained by him, the gentleman doesn't seem to have a lot of experience with the Infrastructure Operations, ITSM and ITIL, but he gives his perspective as developer. Fair enough!
You're a wizard, Don. Nice one!
hello Adam, get back to me, i want to discuss business with you
Am I hearing it wrong about “deployment frequency” definition at :54 sec ? It is number of successful deployments not “ change commit to production “.
Yup, I misspoke and somehow I missed it later. You are correct.
That's a pretty cool lamp! 😉
For the record, that 3d-printed Sleuth logo in the background was built by Andy and he did a kick ass job :)
Great content with insights
Maybe Habit Teacher could be a headline that could spark some understanding and foster a conversation about what we do. You know I love your videos, but just to iterate. That amount of distance between the camera and you in these videos is the distance you should keep 😆
I like how you had to emphasise that it is "not about the metrics, but about the developers". Obviously just another way to micromanage developers and have micro level insights, so this message had to be stressed, kind off like propaganda brainwashing. Not a sincere product definitely.
100% agreed with @JonathanCrossland and his comment - it is not about developers, but about desire to do more quicker, which in a long term is a recipe for disaster.
Yo these camera angles are leet.
THis is all just words which I could have read on any website. Nothing is shown. How to capture the metrics is shown which is the main thing.
❤
There are so many bad assumptions here.
These metrics do not show anything to developers. Every time you sample these metrics the context is different. Different work, task, code, requirement etc.
Shipping more and smaller is wrong.
Ship well and when necessary is more correct.
Small changes can easily affect a lot of things. The assumption here is that a small change is small. Sometimes a small change is huge.
It also has nothing to do with quicker. Being fast is nonsense for developers, it is a business desire. We should stop trying to apply business desires as a positive to developers.
Change failure rate is bad, because a failure ALWAYS changes.the nature of a failure is always different, in how you measure and how it works.
I think both things can be true. The State of DevOps research showed by surveying 30+k companies that deploying frequently did strongly correlate to organizations achieving their business goals, but as you correctly point out, at the micro it is a lot more complex. I noticed you have a TH-cam dev channel, would you be up for a collab/debate/discussion/whatever? I'd love to chat further :)
Context matters here. The goal is to be capable of releasing software on demand. This typically means at the tempo of the business value proposition(features and bug fixes). The DORA metrics align to this concept, just balance them against the deployment cadence needed to achieve business value.
@@donbrown573 This is basic correlation/causation cargo cult fallacy. Just because successful people disproportionately own BMWs does not mean owning a BMW will make you successful.
DORA Metrics are metrics, can be misunderstood and misused, leadership as usual look at numbers, but the numbers doesn’t explain the story or the research and all the dependencies that caused these numbers.. and sooner or later, these metrics will be used to measure performance, and payout bonuses, it’s going to be a mess..
just my opinion
so what do you do with the metrics? "Hey Bob, start programming faster!" lol
just another system to make software development more costly and convulated. Great for job security, managers hate this. Just makes devs want EASIER tasks, so they ship "faster". While poor junior get guys huge complex feature that takes forever.
So those metrics tell me nothing. Just give me css changes to make those metrics awesome.
We see DORA metrics most useful at a team or service level, and not at an individual level. A team, looking to improve their performance, may see a long review lag time and decide to change their process on PR reviews. A team seeing a high change failure rate may decide to increase deployments or invest in better CI testing.
I agree trying to apply these metrics to a single developer, or worse, use them to compare devs with no other context is very harmful.