Bayesian Sequentially Monitored Multi-arm Experiments with Multiple Comparison Adjustments

Abstract: Randomized experiments play a major role in data-driven decision making across many different fields and disciplines. In medicine, for example, randomized controlled trials (RCTs) are the backbone of clinical trial methodology for testing the efficacy of new drugs and therapies versus existing treatments or placebo. In business and marketing, randomized experiments are typically referred to as A/B tests when there are only two arms, or variants, in the experiment, and as multivariate A/B tests when there are more than two arms. Typical applications of A/B tests include comparing the effectiveness of different ad campaigns, evaluating how people respond to different website layouts, or comparing different customer subpopulations to each other. This paper focuses on multivariate A/B testing from a digital marketing perspective, and presents a method for the sequential monitoring of such experiments while accounting for the issue of multiple comparisons. In adapting and combining the methods of two previous works, the method presented herein is straightforward to implement using standard statistical software and performs quite well in various simulation studies, exhibiting better power and smaller average sample sizes than comparable methods.