I'm a Scrum Master on two teams. On one of them, we did include "Must pass User Acceptance Testing" as part of our Definition of Done. This was effective in illuminating the problems involving a hand-off in your work to get work items to Done. Adding UAT to our Def of Done was not effective in solving the problem(s) that it exposed. We still had a time gap in waiting for someone outside the team to do UAT on the work, before we called it Done. The main effect to our workflow is we had more "carryover" from Sprint to Sprint, i.e., it made the issue of the UAT hand-off far more transparent.
Love the approach of adding this within the Definition of Done, really helps it become re-emphasize the importance of building "Quality" from the start. Also stating that UAT is an anti pattern would help with trying to add this into the DOD as an "Experiment" so at least teams can give it a try. Going to add this to my list, another great listen!
I am glad I saw this one. I didn’t realize that my new organization may be doing UAT which may be why the coach suggested that UAT could be added to the Definition of Done. I will have to check when one of the projects finally starts. For a previous organization, we didn’t use UAT. I was fortunate in that the lead Technical person had worked with Agile before coming to the team (when he started was before my time). However, he was was constantly testing. He even helped the developers come up with better tests. That was a major factor in our product being really good. I suspect that as testing automation increased, he would not have had to spend as much time testing. That said, we did have some customers finally come out to see the product. They took so long to get out there, that the Tech Lead sold them on it was more for training than getting feedback. Fortunately, the Tech Lead (in some ways) was a proxy customer even though his knowledge was a bit old.
Perfectly agree with Todd to include UAT as part of the Definition of Done. There are a lot of reasons why stakeholders can’t get rid of UAT, one of which is they want a hands-on experience first before we deploy it to PROD (performing end to end testing with that feature which has an impact on the overall user journey). Another reason is that they have quality issues (PROD bugs) so they want to make sure it goes smoothly (sounds we have to tackle quality here). This is a tough one to overcome.
Very good video. Here value matters than the process. The value here that the team somehow need to get a quick feedback from customer/stakeholders before end of the Sprint. Teams should discuss this point with their stakeholders and customize the way they can achieve it.
DoD proposal is great, just do: "Business people and developers must work together daily throughout the project." and the issue is solved, right? Plan B: What about the Product Owners on Scrum Teams? Can't they validate whether what was build fits (or might fit) the needs of the users? Or is that just wishful thinking? Is there an episode on the phrase "works as designed"? drives me crazy... Love the show! Keep it up!
It is an interesting question. Maybe the real problem is that in Agile, you shouldn't rely on end users to help test the story? I'm not sure. My personal paradigm is that I was trained in a waterfall landscape at the beginning of my career so it is admittedly difficult to look beyond what I've learned. We do a UAT phase after a sprint but it runs concurrently with the next sprint in the cycle so while we're running a sprint, we're doing UAT on the stories completed in the previous sprint. I will say that a majority of the issues we find a UAT are because the story wasn't refined enough so when the users sit down to test, they start to see the changes enacted by the story in the context of the bigger picture and then either ask for additional changes or maybe refine their vision of the story a bit more. In a way, it does sortof feel like a crutch that enables poor story refinement but on the other hand, sometimes you don't know what you don't know. We had considered asking the users to test in the sprint but we sometimes don't have stories ready until the very end of the sprint which makes it hard to business users to schedule testing. For us, it seems like running a planned UAT aligns better with the customer's expectations, unless we tell them they don't need to test, which again exposes the lack of refinement in the stories.
Thanks for the comment! When you say "a majority of the issues we find a UAT are because the story wasn't refined enough" perhaps you could find a way to adapt your refinement practices? It might also be worthy of an experiment asking users to test during a Sprint. Because you don't know what you don't know try to involve users sooner. Waiting for a Sprint Review or a separate UAT phase is costly.
@@AgileforHumans We'll have to think about that. The refinement of stories is a challenge for us. Maybe that's a good future topic. How detailed should a story be? What's a good story look like?
Great questions to ask the team! And maybe, it's okay if they aren't perfect going into a Sprint and you get the chance to talk to your users as you build them.
We still have that in our Product Backlog. The order dropped a bit as we have been focusing our time on content. There will be a day when we do have merch 😊
Dudes, I work for one of the top three Banks in the country. How does one do UAT in the sprint if you are always adding from the dev environment to the UAT environment? I contend if we have enterprise releases, you need to have a fully complete UAT environment BEFORE UAT can start. Thoughts?
Hi! What does it take to have a fully complete UAT environment? Often, what we see when we hear that is tightly coupled architecture which doesn't allow different products to deploy independently. In that case, we start to promote the idea of investing in modern infrastructure/architecture. When a developer checks in code, we want it to hit a CI/CD pipeline that has automated testing that, once passing, elevates and deploys it into upper-level environments (such as UAT).
i have a problem with todd's idea, if we add to the definition of done that someone outside the scrum team should test the increment before it's done, aren't we removing from the scrum team the capability of creating a done increment by itself? shouldn't the scrum team itself "have all the necessary skills to create a done increment?", shouldn't this person be in the scrum team?. The products in which i work don't have an specific client that demands to approve some specific feature, the features are meant to be available to a lot of different clients, so we usually don't receive this kind of pressure, but we have some QA inside the scrum team, they usually just test what others coded and help they debug, they don't code, but if they need to code, they ask someone else to test what they coded, so there's always an "unbiased" final testing before moving to done.
Perhaps my (Todd) discussion point wasn't quite clear. I was not referring to an approval by a user. Rather, I was promoting the idea that we involve users/stakeholders while we do the work. Why even wait for a Sprint Review? Adding "showed it to a user(s)" to the definition of "done" seems like a great way to ensure user/stakeholder involvement early and often. You could also add "showed it to someone outside of the Scrum Team" just to get an outside perspective and have others ask question.
Perhaps Barry Overeem should add a 9th Scrum Master stance of "Phase Gate Buster" with a Peter Venkman mascot? Since you asked for intro music suggestions, I personally favor the following... th-cam.com/video/lhg9bYNLvOg/w-d-xo.html
guys so please explain how do you perform E2E testing. Imagine you have 10 scrum teams collaboratively creating a solution 2 of them come from the external supplier, and the entire solution is made up of 2 different technologies. How can you avoid an E2E test phase before releasing it?
How does Google, Amazon, Facebook, etc.. do it? 😊 They do it by paying major attention to architecture and infrastructure along with investments in test automation. That may seem overwhelming we totally get it. Work with the Scrum Teams to make the definition of "done" more stringent over time. Take one step next Sprint! Make the DoD your focus of the next Sprint Retrospective. Get 15% closer to those big companies a month from now. It's possible!
@@AgileforHumans How to manage the architecture in Scrum? If it is a large application, an architect is looking at the larger vision (technically) across multiple teams and needs to stay ahead of the developers who are working in a sprint.
This was so confusing to me, until I realized that for maybe most of your audience, "UAT" is a thing that happens AFTER the sprint. What the hell, why? I know my career experience is limited, but I struggled to figure out what you guys were talking about. I had POs do their UAT in the Dev or Staging environment immediately after the QA stage in our Kanban board, before the work was deployed.. so, there was no phase gate in my process, right? In reading the comments, I'm also surprised to read about a "UAT Environment".. is this what some people call Staging, or is it a whole different thing just for POs? Anyway, here's my current understanding of your topic: people are creating increments that aren't releaseable because there's a POST-SPRINT UAT requirement, and that constitutes a phase gate. And that's an anti-pattern. (And, I don't need to worry, because I do UAT during the sprint..)
It is actually good to hear that it was confusing (in some way). That means that we are hopefully getting rid of this ridiculous practice. You are correct in that UAT is often viewed as a stage gate to production. So teams that are doing this are not delivering "done" Increments. What you are describing is what we expect. That UAT is done when a PBI is "done" during a Sprint.
Here are my obstacles: -Organization has a gated release process for Test, Stage and Prod. Fixes and enhancements are released to UAT/Stage following QA sign off in Test. -Unless there is an emergency, updates and fixes are only released to Stage on Tuesdays and Thursdays. Org is migrating to CICD. -UAT is completed by individuals with other responsibilities; and UAT within my organization is 95% manual. We also lack automation for QAT. With the current constraints, my team considers a story done when QA signs off and release notes are completed/approved.
I'm a Scrum Master on two teams. On one of them, we did include "Must pass User Acceptance Testing" as part of our Definition of Done. This was effective in illuminating the problems involving a hand-off in your work to get work items to Done. Adding UAT to our Def of Done was not effective in solving the problem(s) that it exposed. We still had a time gap in waiting for someone outside the team to do UAT on the work, before we called it Done. The main effect to our workflow is we had more "carryover" from Sprint to Sprint, i.e., it made the issue of the UAT hand-off far more transparent.
Love the approach of adding this within the Definition of Done, really helps it become re-emphasize the importance of building "Quality" from the start. Also stating that UAT is an anti pattern would help with trying to add this into the DOD as an "Experiment" so at least teams can give it a try.
Going to add this to my list, another great listen!
I am glad I saw this one. I didn’t realize that my new organization may be doing UAT which may be why the coach suggested that UAT could be added to the Definition of Done. I will have to check when one of the projects finally starts.
For a previous organization, we didn’t use UAT. I was fortunate in that the lead Technical person had worked with Agile before coming to the team (when he started was before my time). However, he was was constantly testing. He even helped the developers come up with better tests. That was a major factor in our product being really good. I suspect that as testing automation increased, he would not have had to spend as much time testing. That said, we did have some customers finally come out to see the product. They took so long to get out there, that the Tech Lead sold them on it was more for training than getting feedback. Fortunately, the Tech Lead (in some ways) was a proxy customer even though his knowledge was a bit old.
Perfectly agree with Todd to include UAT as part of the Definition of Done. There are a lot of reasons why stakeholders can’t get rid of UAT, one of which is they want a hands-on experience first before we deploy it to PROD (performing end to end testing with that feature which has an impact on the overall user journey). Another reason is that they have quality issues (PROD bugs) so they want to make sure it goes smoothly (sounds we have to tackle quality here). This is a tough one to overcome.
Very good video. Here value matters than the process. The value here that the team somehow need to get a quick feedback from customer/stakeholders before end of the Sprint. Teams should discuss this point with their stakeholders and customize the way they can achieve it.
DoD proposal is great, just do: "Business people and developers must work together daily throughout the project." and the issue is solved, right? Plan B: What about the Product Owners on Scrum Teams? Can't they validate whether what was build fits (or might fit) the needs of the users? Or is that just wishful thinking? Is there an episode on the phrase "works as designed"? drives me crazy... Love the show! Keep it up!
It is an interesting question. Maybe the real problem is that in Agile, you shouldn't rely on end users to help test the story? I'm not sure. My personal paradigm is that I was trained in a waterfall landscape at the beginning of my career so it is admittedly difficult to look beyond what I've learned.
We do a UAT phase after a sprint but it runs concurrently with the next sprint in the cycle so while we're running a sprint, we're doing UAT on the stories completed in the previous sprint. I will say that a majority of the issues we find a UAT are because the story wasn't refined enough so when the users sit down to test, they start to see the changes enacted by the story in the context of the bigger picture and then either ask for additional changes or maybe refine their vision of the story a bit more. In a way, it does sortof feel like a crutch that enables poor story refinement but on the other hand, sometimes you don't know what you don't know.
We had considered asking the users to test in the sprint but we sometimes don't have stories ready until the very end of the sprint which makes it hard to business users to schedule testing. For us, it seems like running a planned UAT aligns better with the customer's expectations, unless we tell them they don't need to test, which again exposes the lack of refinement in the stories.
Thanks for the comment! When you say "a majority of the issues we find a UAT are because the story wasn't refined enough" perhaps you could find a way to adapt your refinement practices? It might also be worthy of an experiment asking users to test during a Sprint. Because you don't know what you don't know try to involve users sooner. Waiting for a Sprint Review or a separate UAT phase is costly.
@@AgileforHumans We'll have to think about that. The refinement of stories is a challenge for us. Maybe that's a good future topic. How detailed should a story be? What's a good story look like?
Great questions to ask the team! And maybe, it's okay if they aren't perfect going into a Sprint and you get the chance to talk to your users as you build them.
What is the best way to manage "UAT" when you are working on the MVP?
th-cam.com/video/FnHAlRde0fk/w-d-xo.html
Ryan & Todd,
Any update on how to get/order those merchandise (hoodies) ?
We still have that in our Product Backlog. The order dropped a bit as we have been focusing our time on content. There will be a day when we do have merch 😊
Is UAT same as sprint review?
Nope. UAT is an anti-pattern: th-cam.com/video/FnHAlRde0fk/w-d-xo.html
Dudes, I work for one of the top three Banks in the country. How does one do UAT in the sprint if you are always adding from the dev environment to the UAT environment? I contend if we have enterprise releases, you need to have a fully complete UAT environment BEFORE UAT can start. Thoughts?
Hi! What does it take to have a fully complete UAT environment? Often, what we see when we hear that is tightly coupled architecture which doesn't allow different products to deploy independently. In that case, we start to promote the idea of investing in modern infrastructure/architecture. When a developer checks in code, we want it to hit a CI/CD pipeline that has automated testing that, once passing, elevates and deploys it into upper-level environments (such as UAT).
i have a problem with todd's idea, if we add to the definition of done that someone outside the scrum team should test the increment before it's done, aren't we removing from the scrum team the capability of creating a done increment by itself? shouldn't the scrum team itself "have all the necessary skills to create a done increment?", shouldn't this person be in the scrum team?. The products in which i work don't have an specific client that demands to approve some specific feature, the features are meant to be available to a lot of different clients, so we usually don't receive this kind of pressure, but we have some QA inside the scrum team, they usually just test what others coded and help they debug, they don't code, but if they need to code, they ask someone else to test what they coded, so there's always an "unbiased" final testing before moving to done.
Perhaps my (Todd) discussion point wasn't quite clear. I was not referring to an approval by a user. Rather, I was promoting the idea that we involve users/stakeholders while we do the work. Why even wait for a Sprint Review?
Adding "showed it to a user(s)" to the definition of "done" seems like a great way to ensure user/stakeholder involvement early and often. You could also add "showed it to someone outside of the Scrum Team" just to get an outside perspective and have others ask question.
Agree, we should avoid blocking dependencies as much as we can, but we can have platform teams with high level of automation and self service approach
Perhaps Barry Overeem should add a 9th Scrum Master stance of "Phase Gate Buster" with a Peter Venkman mascot?
Since you asked for intro music suggestions, I personally favor the following...
th-cam.com/video/lhg9bYNLvOg/w-d-xo.html
"I ain't afraid of doing Scrum"
guys so please explain how do you perform E2E testing. Imagine you have 10 scrum teams collaboratively creating a solution 2 of them come from the external supplier, and the entire solution is made up of 2 different technologies. How can you avoid an E2E test phase before releasing it?
How does Google, Amazon, Facebook, etc.. do it? 😊 They do it by paying major attention to architecture and infrastructure along with investments in test automation.
That may seem overwhelming we totally get it. Work with the Scrum Teams to make the definition of "done" more stringent over time. Take one step next Sprint! Make the DoD your focus of the next Sprint Retrospective. Get 15% closer to those big companies a month from now. It's possible!
@@AgileforHumans How to manage the architecture in Scrum? If it is a large application, an architect is looking at the larger vision (technically) across multiple teams and needs to stay ahead of the developers who are working in a sprint.
@@stevecomenzo2121 check Team Topologies concept, that might help you a lot 🙂
This was so confusing to me, until I realized that for maybe most of your audience, "UAT" is a thing that happens AFTER the sprint. What the hell, why? I know my career experience is limited, but I struggled to figure out what you guys were talking about. I had POs do their UAT in the Dev or Staging environment immediately after the QA stage in our Kanban board, before the work was deployed.. so, there was no phase gate in my process, right? In reading the comments, I'm also surprised to read about a "UAT Environment".. is this what some people call Staging, or is it a whole different thing just for POs? Anyway, here's my current understanding of your topic: people are creating increments that aren't releaseable because there's a POST-SPRINT UAT requirement, and that constitutes a phase gate. And that's an anti-pattern. (And, I don't need to worry, because I do UAT during the sprint..)
It is actually good to hear that it was confusing (in some way). That means that we are hopefully getting rid of this ridiculous practice. You are correct in that UAT is often viewed as a stage gate to production. So teams that are doing this are not delivering "done" Increments.
What you are describing is what we expect. That UAT is done when a PBI is "done" during a Sprint.
Here are my obstacles:
-Organization has a gated release process for Test, Stage and Prod. Fixes and enhancements are released to UAT/Stage following QA sign off in Test.
-Unless there is an emergency, updates and fixes are only released to Stage on Tuesdays and Thursdays. Org is migrating to CICD.
-UAT is completed by individuals with other responsibilities; and UAT within my organization is 95% manual. We also lack automation for QAT.
With the current constraints, my team considers a story done when QA signs off and release notes are completed/approved.
I’m a guitar player and can make a better intro track for you :)
Do it!