To answer the audience question. The choice of using a proportions test (z-test) is dependent on the metric. If the metric is a proportion, then it would be inappropriate to use a t-test, which tests for difference in mean. Second, for the A/A test what we care about is the distribution of the p-values. More specifically, we want the distributions to be uniform. This distribution is independent of the sample size under the null hypothesis. If the null hypothesis is false, which means there is something wrong with the randomization, then the sample size will affect the distribution and it would look different, but this doesn't matter because at this point you've clearly failed the A/A.
To answer the audience question. The choice of using a proportions test (z-test) is dependent on the metric. If the metric is a proportion, then it would be inappropriate to use a t-test, which tests for difference in mean. Second, for the A/A test what we care about is the distribution of the p-values. More specifically, we want the distributions to be uniform. This distribution is independent of the sample size under the null hypothesis. If the null hypothesis is false, which means there is something wrong with the randomization, then the sample size will affect the distribution and it would look different, but this doesn't matter because at this point you've clearly failed the A/A.
Related blog: medium.com/netflix-techblog/its-all-a-bout-testing-the-netflix-experimentation-platform-4e1ca458c15