Metric Selection in A/B Testing: Easy Explanation for Data Science Interviews

In this video, we will go over another important step in the process of designing A/B tests: how to select the right metrics in A/B testing? This is the second video of a video series - Top A/B testing questions in Data Science interviews. Here are the topics covered in this video:
- What are driver metrics?
- What are the attributes of driver metrics?
- How to develop and select driver metrics?
Sample size estimation:
• Sample Size Estimation...
🟢Get all my free data science interview resources
www.emmading.com/resources
🟡 Product Case Interview Cheatsheet www.emmading.com/product-case...
🟠 Statistics Interview Cheatsheet www.emmading.com/statistics-i...
🟣 Behavioral Interview Cheatsheet www.emmading.com/behavioral-i...
🔵 Data Science Resume Checklist www.emmading.com/data-science...
✅ We work with Experienced Data Scientists to help them land their next dream jobs. Apply now: www.emmading.com/coaching
// Comment
Got any questions? Something to add?
Write a comment below to chat.
// Let's connect on LinkedIn:
/ emmading001
====================
Contents of this video:
====================
00:00 Introduction
00:12 Metric selection in A/B tests
00:51 What are driver metrics?
01:35 What are the attributes of driver metrics?
03:58 How to select driver metrics?
04:46 Fully understand the goal of a test
06:04 Analyze user experience
08:15 Summary

[@sharonyang5354]

Hey Emma, in this case would 1. time spend per video per creators, and 2. the average number of dislikes for smaller creators, are both metrics we should have in the experiment, or only one of them?

[@zhaoyanzhi741]

Hi Emma! Thank you so much for sharing metrics related topics which is super helpful! I know that you once talked about guardrail metrics in other A/B testing video, could you also make one specific video talking about Guardrail metrics:)

[@emma_ding]

Great suggestion Zhaoyan! I am always looking for new ideas for content. Stay tuned!

[@yarongsong9566]

@@emma_ding Thanks Emma for the great video! I'm reviewing all your videos carefully to prepare for my recent job interviews. Thanks! Btw, I have a follow up question. For the TH-cam test in this video, why not chose a Guardrail metric as well? We should select at least one counter metric for a A/B test, right? Thx.

[@jamesrobisnon9165]

Great video. In the TH-cam dislike button example, if the goal for removing dislike counts was to protect creators, then why they could still see it in TH-cam studio? Was the goal just to hide it from viewers? It does not make much sense to me what was the purpose of that change

[@cherylf4788]

Your video is always helpful! One question - when defining metrics of A/B testing, should we only mention driver metrics or both driver and success metrics?

[@causalinference4176]

I think the driver metrics in the lecture should be the metrics for experiments, they include the subset of goal, driver, and guardrail metrics

[@user-bn6tc4vv6l]

is there driver metrics = success metrics for experiment? meaning use it to determine whether experiment is successful or not?

[@sanjnasrivatsa3485]

Given all the information for the TH-cam example, could the desired outcome at 6:47 be more engagement from small creators (posting more videos) despite getting dislikes than before the dislike button was hidden? Measuring number of dislikes confused me. Are we then trying to see if by hiding number of dislikes, will we discourage people from randomly disliking videos as a form of harassment?

[@veee9769]

same got confused too, if the treatment doesn't have dislike button, then the avg dislikes ofc will decrease....

[@chessplayer0106]

I only click "like" never "dislike" on your videos Emma!

[@emma_ding]

Thank you Victor!

[@AnShett]

For the last example, would the metric be avg dislikes per viewer or avg dislikes per video per user? The latter seems to capture the use case better. If we pick the latter, will this impact the experiment design?

[@cococnk388]

Average dislike per vidéo of small creators.... i guess????

[@arojitdas8256]

Nice explanation .....

[@emma_ding]

Thank you Arojit! Glad it was helpful!

[@livingbyheart8510]

I think you should have choosen a different example to explain metrics to select other than the youtube one, it would have been more beneficial to understand.