Facebook's A B Platform Interactive Analysis in Realtime - @Scale 2014 - Data

Itamar Rosenn, Engineering Manager at Facebook
This talk presents Facebook¹s platform for streamlined and automated A/B test analysis. We will discuss the system architecture, how it enables interactive analysis on near-realtime results, and the challenges we've faced in scaling the system to meet our customer needs.

[@gordonweiss1372]

Live it, Learn it, Love it.

[@jiahuizhang2158]

One mistake: in Frequentist Inference, 95% confidence interval doesn't mean there is 95% chance the true value would fall into the interval, but if you repeat the estimation many times, there will be about 95% cases that the true value is indeed in the interval.

[@Psychera]

Brilliant catch! I was about to call that out too. But really nice presentation. This guy clearly knows his stuff and has excellent clarity of thoughts.

[@GenzhPuff]

hey quick question, repeat many times, 95% cases in the interval, isn't that the empirical probability of 95%?

[@howardsmith9032]

@@GenzhPuff No the probability the point estimate is in the interval is either 0 or 1.

[@andrepryor2571]

how do you do that?