2000 character limit reached
Best of Three Worlds: Adaptive Experimentation for Digital Marketing in Practice (2402.10870v3)
Published 16 Feb 2024 in cs.LG and stat.ME
Abstract: Adaptive experimental design (AED) methods are increasingly being used in industry as a tool to boost testing throughput or reduce experimentation cost relative to traditional A/B/N testing methods. However, the behavior and guarantees of such methods are not well-understood beyond idealized stationary settings. This paper shares lessons learned regarding the challenges of naively using AED systems in industrial settings where non-stationarity is prevalent, while also providing perspectives on the proper objectives and system specifications in such settings. We developed an AED framework for counterfactual inference based on these experiences, and tested it in a commercial environment.
- Best of both worlds: Stochastic & adversarial best-arm identification. In Conference on Learning Theory. PMLR, 918–949.
- Best arm identification in multi-armed bandits.. In COLT. 41–53.
- Bridging the gap between regret minimization and best arm identification, with application to A/B tests. In The 22nd International Conference on Artificial Intelligence and Statistics. 1988–1996.
- Accurate inference for adaptive linear models. In International Conference on Machine Learning. PMLR, 1194–1203.
- Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research 7, Jun (2006), 1079–1105.
- Adaptive Experimental Design and Counterfactual Inference. In RecSys CONSEQUENCES Workshop.
- Confidence intervals for policy evaluation in adaptive experiments. arXiv preprint arXiv:1911.02768 (2019).
- D. G. Horvitz and D. J. Thompson. 1952. A Generalization of Sampling Without Replacement from a Finite Universe. J. Amer. Statist. Assoc. 47, 260 (1952), 663–685.
- Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics 49, 2 (2021), 1055–1080.
- lil’ucb: An optimal exploration algorithm for multi-armed bandits. In Conference on Learning Theory. 423–439.
- Kevin G Jamieson and Lalit Jain. 2018. A bandit approach to sequential experimental design with false discovery control. Advances in Neural Information Processing Systems 31 (2018), 3660–3670.
- Jimmy Jin and Leio Pekelis. 2018. Acceleration of A/B/n Testing under time-varying signals. (2018).
- Peeking at a/b tests: Why it matters, and what to do about it. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1517–1525.
- On the complexity of best-arm identification in multi-armed bandit models. The Journal of Machine Learning Research 17, 1 (2016), 1–42.
- Trustworthy online controlled experiments: A practical guide to a/b testing. Cambridge University Press.
- Bayesian Meta-Prior Learning Using Empirical Bayes. Management Science 68, 3 (2022), 1737–1755.
- Chao Qin and Daniel Russo. 2022. Adaptivity and confounding in multi-armed bandit experiments. arXiv preprint arXiv:2202.09036 (2022).
- Herbert Robbins. 1970. Statistical methods related to the law of the iterated logarithm. The Annals of Mathematical Statistics 41, 5 (1970), 1397–1409.
- Daniel Russo. 2016. Simple bayesian algorithms for best arm identification. In Conference on Learning Theory. 1417–1418.
- Contextual Multi-Armed Bandits for Causal Marketing. In Workshops of International Conference on Machine Learning (ICML).
- On the bias, risk and consistency of sample means in multi-armed bandits. arXiv preprint arXiv:1902.00746 (2019).
- Inference for batched bandits. Advances in neural information processing systems 33 (2020), 9818–9829.