Not really. For most sites, the main success never changes (revenue per visitor, lead acquisition, etc). You want to make sure you are always testing to your main primary metric. Each test that you run should focus on learning how to increase the primary metric. There are some sites that don't have a single clear primary metric and in that case you would be right to define the test first. In my experience, these types of sites are the rare exception though.
Awesome video, really appreciated. As a designer i've been asked to create variations for A/B testing website layouts without really understanding the scope of the process. Your video really made me curious to what information i can actually extract from putting similar tests in place for my own projects. Cheers to that mate!
A truly great video. Really enjoyed the in-depth explanation and clarity with which each topic within the video was discussed. Thank you for such amazing content. Really appreciate it !!!
I really enjoyed this video and found it very helpful to my new endeavor....I have spent my career driving sales and engagement in brick and mortar stores but now due to the pandemic must pivot to focus more on ecomm. Promoting a tool that integrates A/B testing into a company's site is what I am learning about now and your video really helped me to understand how I can explain the process to others.
I can really get some value out of this! I am into economic psychology/behavioral economics and am about to refine my data skills. Something of great relevance as the intepretation of data becomes more and more meaningful. Your content comes right at the right time, for me personally! Much appreciated. Subscribed!
Thanks for this informative video on A/B testing! I was wondering if one only uses A/B testing to compare only two alternative solutions, e.g. web pages, or if it is also common to run a series of A/B tests to compare a larger number of solutions, e.g. four different web page layouts.
Great question Berra. Yes, you can have as many variations (web pages, elements, etc) that your traffic supports. You could easily run four web pages with different layouts as a single a/b test. Sometimes people call it a/b/n testing because there isn't a limit to the number of variations you can have.
Hi, so one of my questions is 'how do you conduct the test'? Like Under Armor for example, do you just implement the changes on the live site lets say for a day and compare site traffic (before and after), or do you get a group of people and have them interact with both sites and see which site they respond to (an organized/planned testing method)? Your video was very informative and helpful, yet this is something I've been confused on for a while.
Hi Yash. All a/b testing tools allow you to split the traffic so some visitors see the control on the live site and others see the coded variations on the live site. They are randomly put into the test and they don't know they are being tested. This allows you to see causal data on their behavior so you can compare conversion rates to see if the tested variations increased the primary rate you are trying to improve. Also, you would want the test to run fro longer than a day. See this video for how to call a winner and how long to run the test: th-cam.com/video/c5YaKCMraO8/w-d-xo.html
Incredible video! Very clear, well structured and simple, full of insightful definitions! Thank you for sharing this with us. Actually, I am not running any A/B testing, however, I can clearly see the potential behind this test, so I would like to use this test to check my idea for my future start-up. If in the city where I am currently living people are experimenting problems with public transports, and if yes, I would like to know if they would be keen to utilise an alternative method of transport in the city which is the inland waterway taxi boats/bus.
Is there any way you could reveal what bing changed? I’m very curious to what could’ve possibly made them that much more money. Really cool. Love your videos.
I'm not sure. Most of us in the industry do this every day, so we always have numbers we are working with, but they are usually proprietary because they are a client's data. Would it be helpful if you had some made up numbers to work with?
Hello - I'm just beginning to learn about A/B testing and this video has really helped me in learning a lot of the fundamentals about the subject. I do have questions about the actual testing itself. For example if you are doing a test on a landing page, does the Control and Variation get tested at the same time? Meaning some members of the public may click and land on the Control and some may land on the Variation? Or is it done is stages, like first 2 months for Control then next 2 months for Variation? If it is done at the same time, is it possible that one person might click on the control landing page one day then click on the variation on another day?
Welcome to a very exciting field and subject. These are great questions. You always want the traffic to be randomly assigned to each variation during the same time period. This will reduce the amount of time or seasonality variance that would otherwise be introduced if you ran one variation then stopped traffic to it and ran another variation. Testing tools will cookie a visitor into an experience so that they will always see the same variation until you turn off a test. If they use a different device they might see a different experience, but that device will stay with that variation until the end of the test.
Good Info ,so there is no need for a traditional QA team to do these A/B testing ? who is most recommended to do these A/B testing , Developer Or data Analyst or Data Scientist
A cross functional team should support the testing efforts for maximum success. The roles you need to do good testing are a executive sponsor, strategist, analyst, developer, design, QA, project manager. Some roles can be fulfilled by multiple people depending on the size of your testing program. Check out this video: th-cam.com/video/ODSILSud2UQ/w-d-xo.html
Thanks for the video! I was wondering is an A/B testing is possible on large retail touch screens (Kiosk)? Can we use the same softwares used for websites? Thank you
Can anyone recommend a good A/B testing management tool? Something to easily track all the experiments you conducted and their results and dependencies, I'm thinking something similar to Trello, or Jira?
Hi Johnny. I am used Trello in the past and am currently using Jira. Both work well. I have also used Effective Experiments. It is a good all in one tool, but has other drawbacks. I probably depends on what you are looking for in the tool.
I wish it were that easy. I made this video almost two years ago and after the first year it had barely any views. It is being shown more now, but I don't think it is because of how many times I said the word "testing" in the video. Either way, I hope you enjoyed the video. :-)
Usually you are tracking your metrics with an analytics tool or your testing tool. The order value would be passed in the confirmation page and you would use AOV as the primary metric when looking at your results.
UX design and A/B testing go hand in hand. They use each other to get better results. UX design is used to create experience to then be tested to verify if they are good or not.
You can do it offline too. One time we did some testing for a cafeteria where we made 3 different types of discount coupons to see what people preferred. 1. Free fruit with purchase 2. Free fries with purchase 3. Free cookie with purchase We counted the coupons turned in to learn the optimal coupon for this cafeteria.
The truth is A/B testing half the time can be massively flawed. Such as the image case, what time of day were they done at different ones? Maybe the last image was ran during lunch time when people were more relaxed, which virtually had no correlation to the image literally at all. All I can say it companies get hung up on A/B testing really really bad rather than actually using common sense or just flat out pick up a phone or talk to someone. There's a safe combination. Just be careful. You have to look at every single contradicting aspect that goes into them as well.
![[#2024MAMA] BIGBANG (빅뱅) - 뱅뱅뱅 (BANG BANG BANG) + FANTASTIC BABY | Mnet 241123 방송](http://i.ytimg.com/vi/EQsYfiPR9uM/mqdefault.jpg)
I've liked all your a/b testing videos, hope you get more views. This is the best elaborate video i've watched.
Thank you very much! That is a nice compliment.
Great video - Shouldn't you first define the idea first and then the success criteria?
Not really. For most sites, the main success never changes (revenue per visitor, lead acquisition, etc). You want to make sure you are always testing to your main primary metric. Each test that you run should focus on learning how to increase the primary metric. There are some sites that don't have a single clear primary metric and in that case you would be right to define the test first. In my experience, these types of sites are the rare exception though.
This video really enlarge my understanding about a/b testing, it is far more than web optimization
You are exactly right. There is so much more to this discipline than meets the eye. Good luck as you dive into it. :-)
I am not sure if AB testing can help prioritizing the features. But once it is prioritized, it can help up giving a sense of its impact.
Awesome video, really appreciated. As a designer i've been asked to create variations for A/B testing website layouts without really understanding the scope of the process. Your video really made me curious to what information i can actually extract from putting similar tests in place for my own projects. Cheers to that mate!
Glad it was helpful.
I am new to A/B testing and I found this video really good! Thank you!
Glad it was helpful!
Thanks for the comprehensive introduction of A/B Testing. It helps a lot.
You are welcome! I'm glad it was helpful to you.
Complete package....
I want to scale my business & would require customer preferences therefore I want to conduct A/B testing.
Doing A/B testing is a great way to learn what your customers want so you can scale your business. Good luck.
Great video. BTW it is fascinating to see that a little change that has been done to a website can make a huge impact on it...just WOW !!
Very true!
A truly great video. Really enjoyed the in-depth explanation and clarity with which each topic within the video was discussed. Thank you for such amazing content. Really appreciate it !!!
Glad you enjoyed it!
I really enjoyed this video and found it very helpful to my new endeavor....I have spent my career driving sales and engagement in brick and mortar stores but now due to the pandemic must pivot to focus more on ecomm. Promoting a tool that integrates A/B testing into a company's site is what I am learning about now and your video really helped me to understand how I can explain the process to others.
Glad it was helpful to you Lauren. Good luck with your transition.
I can really get some value out of this! I am into economic psychology/behavioral economics and am about to refine my data skills. Something of great relevance as the intepretation of data becomes more and more meaningful. Your content comes right at the right time, for me personally! Much appreciated. Subscribed!
Hi Alex, I am glad it is helpful for you and thanks for subscribing.
i like how you layer your content. It is a very nice and logical approach.
Thanks Akshay. Glad you enjoy it!
Thank you It helped to learn about AB testing!
Glad it was helpful!
Thanks much for the video, explained precisely and easy to consume, cheers!
Glad it was helpful!
Excellent video. Not technical, which is bound to help absolute beginners getting into the heart of the matter. :)
Glad you enjoyed it!
Cool & clear presentation, cheers !
Glad you liked it!
Thanks for this informative video on A/B testing! I was wondering if one only uses A/B testing to compare only two alternative solutions, e.g. web pages, or if it is also common to run a series of A/B tests to compare a larger number of solutions, e.g. four different web page layouts.
Great question Berra. Yes, you can have as many variations (web pages, elements, etc) that your traffic supports. You could easily run four web pages with different layouts as a single a/b test. Sometimes people call it a/b/n testing because there isn't a limit to the number of variations you can have.
Hi, so one of my questions is 'how do you conduct the test'? Like Under Armor for example, do you just implement the changes on the live site lets say for a day and compare site traffic (before and after), or do you get a group of people and have them interact with both sites and see which site they respond to (an organized/planned testing method)? Your video was very informative and helpful, yet this is something I've been confused on for a while.
Hi Yash. All a/b testing tools allow you to split the traffic so some visitors see the control on the live site and others see the coded variations on the live site. They are randomly put into the test and they don't know they are being tested. This allows you to see causal data on their behavior so you can compare conversion rates to see if the tested variations increased the primary rate you are trying to improve.
Also, you would want the test to run fro longer than a day. See this video for how to call a winner and how long to run the test: th-cam.com/video/c5YaKCMraO8/w-d-xo.html
Incredible video! Very clear, well structured and simple, full of insightful definitions! Thank you for sharing this with us. Actually, I am not running any A/B testing, however, I can clearly see the potential behind this test, so I would like to use this test to check my idea for my future start-up. If in the city where I am currently living people are experimenting problems with public transports, and if yes, I would like to know if they would be keen to utilise an alternative method of transport in the city which is the inland waterway taxi boats/bus.
A/B tests can be applied in the physical world as well as the digital.
So where are your recommendations for software to use? I'm a student and need something very simple to use.
Every company should do A/B testing!
Totally agree Aniket!
Is there any way you could reveal what bing changed? I’m very curious to what could’ve possibly made them that much more money. Really cool. Love your videos.
It was a headline change: store.hbr.org/product/the-surprising-power-of-online-experiments/R1705E This HBR article has all the details
Great video! Where can I find datasets to practice on?
I'm not sure. Most of us in the industry do this every day, so we always have numbers we are working with, but they are usually proprietary because they are a client's data. Would it be helpful if you had some made up numbers to work with?
@@TestingTheory Right, that’s what I assumed. That would be great!
Very helful, thank you!
Glad it was helpful!
Thank you for nice presentation. Appriciated!!!!!!!!!!!!!!!!!!
You are welcome!
Hello - I'm just beginning to learn about A/B testing and this video has really helped me in learning a lot of the fundamentals about the subject. I do have questions about the actual testing itself. For example if you are doing a test on a landing page, does the Control and Variation get tested at the same time? Meaning some members of the public may click and land on the Control and some may land on the Variation? Or is it done is stages, like first 2 months for Control then next 2 months for Variation? If it is done at the same time, is it possible that one person might click on the control landing page one day then click on the variation on another day?
Welcome to a very exciting field and subject. These are great questions. You always want the traffic to be randomly assigned to each variation during the same time period. This will reduce the amount of time or seasonality variance that would otherwise be introduced if you ran one variation then stopped traffic to it and ran another variation. Testing tools will cookie a visitor into an experience so that they will always see the same variation until you turn off a test. If they use a different device they might see a different experience, but that device will stay with that variation until the end of the test.
nice intro... can you please make a video on actual implementation of a a/b testing on any freely available sites or scenarios... thanks
Yes, I will put that into the queue of videos.
did they do it?? made a video about implementing ab tests on free services?
Good Info ,so there is no need for a traditional QA team to do these A/B testing ? who is most recommended to do these A/B testing , Developer Or data Analyst or Data Scientist
A cross functional team should support the testing efforts for maximum success. The roles you need to do good testing are a executive sponsor, strategist, analyst, developer, design, QA, project manager. Some roles can be fulfilled by multiple people depending on the size of your testing program. Check out this video: th-cam.com/video/ODSILSud2UQ/w-d-xo.html
Very help! Thanks so much!!
Glad it helped!
Awesome, thank you!
You're welcome!
Thanks for the video! I was wondering is an A/B testing is possible on large retail touch screens (Kiosk)? Can we use the same softwares used for websites? Thank you
Yes you can! As long as a Kiosk can have the testing SDK installed or is a web platform you can test on it.
Only 359 sub? With quality content like these you should have at least 6 digits subs...
Thanks I appreciate that. Hopefully that will come :-)
Amazing. thanks
Glad you liked it!
What was Bing's AB test feature!!! The search bar?
It was a headline change: store.hbr.org/product/the-surprising-power-of-online-experiments/R1705E
How to select optimal tools for A/B Testing?
This video might help you out: th-cam.com/video/SFHjSu4XVTs/w-d-xo.html
Can anyone recommend a good A/B testing management tool? Something to easily track all the experiments you conducted and their results and dependencies, I'm thinking something similar to Trello, or Jira?
Hi Johnny. I am used Trello in the past and am currently using Jira. Both work well. I have also used Effective Experiments. It is a good all in one tool, but has other drawbacks. I probably depends on what you are looking for in the tool.
@@TestingTheory thanks a lot
great video
Glad you enjoyed it
He used the word "Testing" so many times so that the algorithm picks up his videos
I wish it were that easy. I made this video almost two years ago and after the first year it had barely any views. It is being shown more now, but I don't think it is because of how many times I said the word "testing" in the video. Either way, I hope you enjoyed the video. :-)
how do you setup A/B testing to measure AOV impact. this video is not covering the major aspects on actually setting up and interpretation of results.
Usually you are tracking your metrics with an analytics tool or your testing tool. The order value would be passed in the confirmation page and you would use AOV as the primary metric when looking at your results.
Is A/B testing similar to UX design?
UX design and A/B testing go hand in hand. They use each other to get better results. UX design is used to create experience to then be tested to verify if they are good or not.
Does a/b testing have to be online only?
You can do it offline too. One time we did some testing for a cafeteria where we made 3 different types of discount coupons to see what people preferred.
1. Free fruit with purchase
2. Free fries with purchase
3. Free cookie with purchase
We counted the coupons turned in to learn the optimal coupon for this cafeteria.
The truth is A/B testing half the time can be massively flawed. Such as the image case, what time of day were they done at different ones? Maybe the last image was ran during lunch time when people were more relaxed, which virtually had no correlation to the image literally at all. All I can say it companies get hung up on A/B testing really really bad rather than actually using common sense or just flat out pick up a phone or talk to someone. There's a safe combination. Just be careful. You have to look at every single contradicting aspect that goes into them as well.
Check your audio levels. it clips very much, it is quite annoying.
Thanks for the feedback
Hey! I know you....:)
UA's recommended section is gone
Interesting, it has been there for years, so they must have just recently changed it. I hope they did some follow up testing to validate that. :-)
Big eyebrows mate!
All the better to express something with. :-)