Thanks for the walkthrough. I have a question: What happens when there are 2 (or more campaigns) for Purchasers_A during the post period? How would this be accounted for in CausalImpact? Example: Pre period: Jan 1 - Jun 30 [no campaign] Post period for Campaign 1 for Audience A: Jul 1 - Dec 31 Post period for Campaign 2 for Audience A: Sep 1 - Sep 30
Since two campaigns adds noise, we would need to look at test and control groups. In the situation that you describe, you would want to find a control and test group. Example, let's say you are testing Facebook traffic: Test group #1 - people from campaign one Test group #2 - people from campaign one Control Group - people that came from FB but not from either campaign. Then you can compare the metric of Test Groups vs Control Group to see if the campaigns had an impact. Just need to make sure that the groups are closely correlated. This is good article on determining if campaigns are effective. towardsdatascience.com/how-can-i-tell-if-my-marketing-campaign-is-working-41cbf5c7dbc6
A causal impact analysis is used to measure the impact of a specific event or treatment. In this case, it’s measuring the impact of media on driving purchases/conversions. It is not intended to evaluate the cost of media on driving an outcome, assuming that is what you are asking. There are other analysis that can help evaluate the impact of spend on driving purchases, most notable analyses tied to ROAS or CAC.
I believe it's not metrics you need, but you need counterfactual populations that did not get affected by the marketing campaign. If the marketing campaign affected the whole population then causal impact is probably not recommended.
Hey great video ! Really appreciate the content. So if i understand correctly, the col A was treatment group, B and C column were control groups, but what are the covariates/independent variables here? I came from the google presentation by Kay on causalimapct package and it discusses abt these independent features. I am afraid i couldnt figure that out here
Great question! You're correct in that the column purchases_A is the treatment group and purchases_B & purchases_C are the control groups. To answer your question, we have to think about what it is that we're trying to predict. We're trying to predict what purchases_A would look like if the treatment was never applied. To do that, we build a model based on the relationship between purchases_A and purchases_B/C during the pre-period, and then we use that model to predict what the treatment group would look like in the post-period if no treatment were ever applied. Then the CausalImpact tool determines the differences between the observed purchases_A and the predicted purchases_A. So in this case, our independent variables/covariates are purchases_B and purchases_C, and the dependent variable is purchases_A.
thank you so much for such detailed walkthrough how to read result, that is rare when ppl try to understand this package
Thanks for the walkthrough. I have a question: What happens when there are 2 (or more campaigns) for Purchasers_A during the post period? How would this be accounted for in CausalImpact?
Example:
Pre period: Jan 1 - Jun 30 [no campaign]
Post period for Campaign 1 for Audience A: Jul 1 - Dec 31
Post period for Campaign 2 for Audience A: Sep 1 - Sep 30
Since two campaigns adds noise, we would need to look at test and control groups.
In the situation that you describe, you would want to find a control and test group. Example, let's say you are testing Facebook traffic:
Test group #1 - people from campaign one
Test group #2 - people from campaign one
Control Group - people that came from FB but not from either campaign.
Then you can compare the metric of Test Groups vs Control Group to see if the campaigns had an impact. Just need to make sure that the groups are closely correlated.
This is good article on determining if campaigns are effective.
towardsdatascience.com/how-can-i-tell-if-my-marketing-campaign-is-working-41cbf5c7dbc6
if you would have ran campaigns for all 3 groups could you factor in spends data?
A causal impact analysis is used to measure the impact of a specific event or treatment. In this case, it’s measuring the impact of media on driving purchases/conversions. It is not intended to evaluate the cost of media on driving an outcome, assuming that is what you are asking. There are other analysis that can help evaluate the impact of spend on driving purchases, most notable analyses tied to ROAS or CAC.
What if you don't have any other metrics that weren't affected, can you still run the test?
I believe it's not metrics you need, but you need counterfactual populations that did not get affected by the marketing campaign. If the marketing campaign affected the whole population then causal impact is probably not recommended.
Hey great video ! Really appreciate the content. So if i understand correctly, the col A was treatment group, B and C column were control groups, but what are the covariates/independent variables here? I came from the google presentation by Kay on causalimapct package and it discusses abt these independent features. I am afraid i couldnt figure that out here
Great question! You're correct in that the column purchases_A is the treatment group and purchases_B & purchases_C are the control groups.
To answer your question, we have to think about what it is that we're trying to predict. We're trying to predict what purchases_A would look like if the treatment was never applied. To do that, we build a model based on the relationship between purchases_A and purchases_B/C during the pre-period, and then we use that model to predict what the treatment group would look like in the post-period if no treatment were ever applied. Then the CausalImpact tool determines the differences between the observed purchases_A and the predicted purchases_A. So in this case, our independent variables/covariates are purchases_B and purchases_C, and the dependent variable is purchases_A.
@@partandsum Amazing explanation ! Thanks so much.