Panos Toulis & W. Guo: ML-assisted Randomization Tests for Complex Treatment Effects in A/B Expts

Subscribe to the channel to get notified when we release a new video.
Like the video to tell TH-cam that you want more content like this on your feed.
See our website for future seminars: sites.google.c...
Tuesday, December 10, 2024: Panos Toulis and Wenxuan Guo (University of Chicago)
Title: ML-assisted Randomization Tests for Complex Treatment Effects in A/B Experiments
Discussant: Xinran Li (University of Chicago)
Abstract: Experimentation is widely used for causal inference and data-driven decision making across disciplines. In an A/B experiment, for example, a business randomizes two different treatments (e.g., website designs) to their customers and then aims to infer which treatment is better. In this paper, we construct randomization tests for complex treatment effects, including heterogeneity and interference. A key feature of our approach is the use of flexible machine learning (ML) models, where the ANOVA-like test statistic is defined as the difference between the cross-validation errors from two ML models, one including the treatment variable and the other without it. This approach combines the predictive power of modern ML tools with the finite-sample validity of the randomization framework, enabling a robust and efficient way to perform causal inference in experimental settings. We demonstrate this combined benefit both theoretically and empirically through applied examples.