Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years, Ron Kohavi 20151130

Ron Kohavi, Distinguished Engineer & General Manager Analysis & Experimentation, Microsoft
The Internet provides developers of connected software, including web sites, applications, and devices, an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using trustworthy controlled experiments (e.g., A/B tests and their generalizations). From front-end user interface changes to backend recommendation systems and relevance algorithms, from search engines (e.g., Google, Microsoft’s Bing, Yahoo) to retailers (e.g., Amazon, eBay, Netflix, Etsy) to social networking services (e.g., Facebook, LinkedIn, Twitter) to Travel services (e.g., Expedia, Airbnb, Booking.com) to many startups, online controlled experiments are now utilized to make data-driven decisions at a wide range of companies. While the theory of a controlled experiment is simple, and dates back to Sir Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, the deployment and mining of online controlled experiments at scale (e.g., hundreds of experiments run every day at Bing) and deployment of online controlled experiments across dozens of web sites and applications has taught us many lessons. We provide an introduction, share real examples, key lessons, and cultural challenges.
Speaker Bio
Ronny Kohavi is a Microsoft Distinguished Engineer and the General Manager for Microsoft's Analysis and Experimentation team at Microsoft's Applications and Services Group. He was previously Partner Architect at Bing, part of the Online Services Division at Microsoft. He joined Microsoft in 2005 and founded the Experimentation Platform team in 2006. He was previously the director of data mining and personalization at Amazon.com, and the Vice President of Business Intelligence at Blue Martini Software, which went public in 2000, and later acquired by Red Prairie. Prior to joining Blue Martini, Kohavi managed MineSet project, Silicon Graphics' award-winning product for data mining and visualization. He joined Silicon Graphics after getting a Ph.D. in Machine Learning from Stanford University, where he led the MLC++ project, the Machine Learning library in C++ used in MineSet and at Blue Martini Software. Kohavi received his BA from the Technion, Israel. He was the General Chair for KDD 2004, co-chair of KDD 99's industrial track with Jim Gray, and co-chair of the KDD Cup 2000 with Carla Brodley. He was an invited speaker at the National Academy of Engineering in 2000, a keynote speaker at PAKDD 2001, an invited speaker at KDD 2001's industrial track, a keynote speaker at EC 10 (2010), a keynote speaker at Recsys 2012, a keynote speaker at Emetrics 2014, and a keynote speaker at KDD 2015. His papers have over 27,000 citations and three of his papers are in the top 1,000 most-cited papers in Computer Science. His Twitter handle is @ronnyK
www.meetup.com/...
www.sfbayacm.or...
bit.ly/KDD2015K...

[@vpdabholkar]

Good introduction to A/B testing along with industry examples from Microsoft and Amazon. Looking forward to reading Ron's book. Thanks for sharing this talk.