From Augmentation to Decomposition: A New Look at CUPED in 2023
Abstract: Ten years ago, CUPED (Controlled Experiments Utilizing Pre-Experiment Data) mainstreamed the idea of variance reduction leveraging pre-experiment covariates. Since its introduction, it has been implemented, extended, and modernized by major online experimentation platforms. Many researchers and practitioners often interpret CUPED as a regression adjustment. In this article, we clarify its similarities and differences to regression adjustment and present CUPED as a more general augmentation framework which is closer to the spirit of the 2013 paper. We show that the augmentation view naturally leads to cleaner developments of variance reduction beyond simple average metrics, including ratio metrics and percentile metrics. Moreover, the augmentation view can go beyond using pre-experiment data and leverage in-experiment data, leading to significantly larger variance reduction. We further introduce metric decomposition using approximate null augmentation (ANA) as a mental model for in-experiment variance reduction. We study it under both a Bayesian framework and a frequentist optimal proxy metric framework. Metric decomposition arises naturally in conversion funnels, so this work has broad applicability.
- Soren Asmussen and Peter Glynn. 2008. Stochastic Simulation. Springer-Verlag.
- Deep Dive Into Variance Reduction. https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/deep-dive-into-variance-reduction/.
- Variance Reduction Using In-Experiment Data: Efficient and Targeted Online Measurement for Sparse and Delayed Outcomes. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3937–3946.
- Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the 6th ACM WSDM Conference. 123–132.
- Zero to hero: Exploiting null effects to achieve variance reduction in experiments with one-sided triggering. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining. 823–831.
- Machine Learning for Variance Reduction in Online Experiments. arXiv preprint arXiv:2106.07263 (2021).
- Ying Jin and Shan Ba. 2023. Toward Optimal Variance Reduction in Online Controlled Experiments. Technometrics 65, 2 (2023), 231–242.
- Winston Lin. 2013. Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. The Annals of Applied Statistics 7, 1 (2013), 295–318.
- Art B Owen. 2013. Monte Carlo theory, methods and examples. (2013).
- Boosted decision tree regression adjustment for variance reduction in online controlled experiments. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 235–244.
- Choosing a Proxy Metric from Past Experiments. arXiv preprint arXiv:2309.07893 (2023).
- Covariate adjustment for two-sample treatment comparisons in randomized clinical trials: A principled yet flexible approach. Statistics in Medicine 27 (2008).
- Huizhi Xie and Juliette Aurisset. 2016. Improving the sensitivity of online controlled experiments: Case studies at netflix. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 645–654.
Paper Prompts
Sign up for free to create and run prompts on this paper.