Bayesian Inference Procedures for A/B Testing: An Overview

Abstract: Bayesian inference for A/B testing is a family of prior and stopping-rule configurations with fundamentally different statistical properties, but it is often discussed as a single method, and no systematic overview exists. This paper organizes common configurations into a three-tier hierarchy: 1) posterior coherence with no error control, 2) false positive rates bounded under continuous monitoring via Bayes factor stopping, and 3) false discovery rate control and calibrated shrinkage via empirical Bayes. Many commercial platforms operate at the lowest tier by default. We show that Bayes factor stopping is near-optimal for a broad class of cost functions, including most proposed in the A/B testing literature; because the same rule also controls the false positive rate, the choice between a decision-theoretic and a frequentist formulation is largely one of parameterization. Furthermore, the empirical Bayes prior is the only path to the third tier, but winner-selected corpora, pooled programs, and heterogeneous metrics can each prevent calibration regardless of corpus size. Simulations against group-sequential and always-valid frequentist baselines show that flat-prior posterior stopping exactly reproduces naive peeking, that a well-calibrated empirical Bayes prior achieves the lowest estimation error, and that expected-loss stopping minimizes regret only when shipping a null-effect variant is nearly free. Error rates, estimation accuracy, and regret are all different risks, and the appropriate method follows from the risks an experimentation program needs to control, not the other way around.

• The paper organizes Bayesian A/B testing into three tiers, showing that posterior rules provide coherence, Bayes factor stopping enables anytime FPR control, and empirical Bayes methods add FDR calibration and shrinkage.
• The paper finds that flat-prior posterior testing reproduces naive peeking with an FPR of 0.303, while proper-prior Bayes factor stopping controls FPR near 0.016–0.019 across tested priors.
• The paper shows that empirical Bayes methods can reduce estimation error and winner’s-curse bias, but their benefits depend on representative historical data, with calibration becoming reliable after roughly 200 experiments and failing under selected or pooled corpora.

Motivation and scope

Bayesian A/B testing is routinely discussed as a single method, but the authors of this overview—members of Spotify's Experimentation Platform team—argue that it is a family of prior and stopping-rule configurations with fundamentally different statistical properties. Depending on configuration, a Bayesian test can reproduce a frequentist z-test exactly, maintain unconditional Type I error control under continuous monitoring, minimize expected launch-decision risk, or calibrate false discovery rates and shrink effect estimates. No systematic comparison of these configurations existed; this paper fills that gap by organizing them into a three-tier guarantee hierarchy and evaluating representative configurations in simulation against group-sequential and always-valid frequentist baselines.

The central thesis is that error rates, estimation accuracy, and regret are distinct risks. The appropriate method follows from which risks an experimentation program needs to control, not from whether the method is labeled Bayesian or frequentist.

Setup

The paper considers a two-group experiment with estimand $\delta = \mu_T - \mu_C$, sequential data accrual, and large-sample normality of the MLE $\hat\delta$ with effective sample size $N_E = n/2$. Three families of stopping rules are analyzed: posterior-based rules (e.g., stop when $P(B > A) > \gamma$), Bayes factor (BF) rules (stop when $\mathrm{BF}_{10}$ exceeds a threshold, with a closed form under the normal-normal conjugate model), and decision-theoretic rules that minimize expected loss over sequential policies.

Two decision-theoretic rules receive particular attention. The expected opportunity loss rule of Stucchio (2015), deployed at VWO and adopted elsewhere, stops when $$\min(E[\max(\delta,0)], E[\max(-\delta,0)]) < \varepsilon$$. Misclassification-cost formulations—with per-observation sampling cost $c$ and error costs $K_I$, $K_{II}$—yield optimal policies that threshold the Bayes factor, via a Bellman argument showing the value function is concave in the posterior probability and terminal costs are affine.

The three-tier hierarchy

Tier 1: posterior coherence. Any proper posterior is a valid belief update regardless of stopping rule (by the Likelihood Principle), but no frequentist error-rate guarantee attaches to decisions based on it. This tier includes both posterior-probability stopping—which has neither error control nor a decision-theoretic justification—and expected-loss stopping, which targets a well-defined loss and is optimal for it when the cost model is accurate. The paper stresses a persistent confusion: the $\alpha$ in $\hat\delta$0 is a Bayesian decision parameter, not the frequentist bound; the two coincide only under a flat prior with fixed-$\hat\delta$1 design.

Tier 2: bounded FPR via Bayes factor stopping. For any proper prior, the BF sequence is a nonnegative martingale under $\hat\delta$2, so Ville's inequality gives $\hat\delta$3 at any stopping time, with no commitment to a maximum sample size. Thresholding on both sides additionally bounds prior-averaged FNR at $\hat\delta$4—though only for paths crossing the lower boundary, and only averaged over the prior, not pointwise in $\hat\delta$5. Without a historical corpus, the recommended default is a JZS Cauchy prior scaled to the minimum detectable effect.

Tier 3: FDR calibration and calibrated shrinkage via empirical Bayes. Adding an empirical Bayes (EB) mixture prior—a point mass at zero plus $\hat\delta$6, fitted to a corpus of past experiments—yields per-metric FDR calibration through the mixture e-process $\hat\delta$7, partial absorption of multiplicity across metrics via the $\hat\delta$8 floor, and shrinkage calibrated to the program's actual effect-size distribution, partially correcting Type M (winner's curse) inflation. Calibration reliability stabilizes around roughly 200 historical experiments per program; correlated metrics within an experiment cannot substitute for more experiments.

A key structural claim is that Tier 3 properties are inseparable from corpus quality. Three failure modes defeat calibration regardless of corpus size: winner-selected corpora drive $\hat\delta$9 and collapse the $N_E = n/2$0 floor entirely; pooled programs with different non-null rates mis-calibrate $N_E = n/2$1 in both directions; heterogeneous metrics pooled into one mixture over-shrink easy-to-move metrics and under-shrink hard ones. None can be fixed by collecting more data of the same kind.

Decision-theoretic optimality of Bayes factor stopping

The paper proves (in an appendix) that for cost functions whose terminal payoffs are affine in the posterior—including misclassification costs and the customer-impact utility of Wan et al. (2023)—the optimal stopping policy is a threshold rule on the Bayes factor. Because the same likelihood-ratio structure makes the BF a martingale under the null, the cost-optimal rule simultaneously carries a Ville-style FPR bound. The practical consequence is strong: the choice between a decision-theoretic and a frequentist formulation is largely one of parameterization. A platform specifying costs implicitly chooses error rates, and vice versa; the Ville-calibrated thresholds are modestly more conservative than the cost-optimal ones (a 25% inflated upper threshold at $N_E = n/2$2, $N_E = n/2$3).

Expected-loss stopping falls outside this class because its criterion involves a nonlinear functional of the posterior. Its stopping boundary converges to zero, eventually triggering at any positive estimate once the posterior concentrates, whereas the BF boundary shrinks as $N_E = n/2$4. Under a flat prior its directional FPR is approximately 0.50—it ships whichever variant is ahead once precision suffices.

Bayesian–frequentist connections

Several equivalences sharpen the argument that framework choice matters less than configuration choice:

• Flat-prior equivalence. Under a flat prior, the posterior mean equals the MLE, $N_E = n/2$5 equals $N_E = n/2$6, credible intervals match confidence intervals, and $N_E = n/2$7 is algebraically identical to $N_E = n/2$8. Flat-prior optional stopping is identical to naive peeking; simulation confirms FPR = 0.303 for both. Several commercial platforms ship flat-prior posterior thresholds as defaults, which the authors characterize as frequentist inference under a different vocabulary.
• mSPRT equivalence. The mixture SPRT statistic is exactly the Bayes factor under the same mixing distribution; choosing the mixing variance is placing a prior on the treatment effect.
• Implicit priors. James–Stein shrinkage and regularized regression adjustment encode prior beliefs without naming them; the frameworks differ in whether prior information is stated explicitly.

Simulation evidence

Four simulations support the taxonomy. In Setting A, flat-prior $N_E = n/2$9 reproduces naive peeking exactly (FPR 0.303), expected-loss stopping yields directional FPR ≈ 0.50, and BF stopping controls FPR at 0.016–0.019 across all proper priors, confirming that the Ville bound is invariant to prior miscalibration while power varies.

Setting B compares valid sequential methods on a common DGP ($P(B > A) > \gamma$0 per arm, non-null rate $P(B > A) > \gamma$1). All control FPR at or below 0.05, but they differ sharply elsewhere:

Method	Config	FPR	Power	Avg. $P(B > A) > \gamma$2	MSE ×10³
GST	OBF, 14 looks	0.051	0.996	430	5.9
Always-valid CS	WS–R	0.013	0.955	392	17.3
BF	Gaussian prior	0.011	0.941	405	3.0
BF	Oracle EB	0.003	0.899	478	2.7
BF	Winner-selected EB	0.013	0.953	388	4.5
BF	Pooled-programs EB	0.007	0.910	432	10.4

Three findings stand out. First, O'Brien–Fleming GST achieves the highest power—even above BF stopping with the oracle EB prior—because its spending function approaches the fixed-test threshold at late looks, while the logarithmically growing BF boundary never crosses on marginal-signal paths; BF detects easy cases fast and misses hard ones. The operational advantage of BF stopping is flexibility: no $P(B > A) > \gamma$3 commitment. Second, the oracle EB prior attains the lowest MSE of any configuration, an advantage attributable to the shrunk posterior mean rather than the stopping rule. Third, misspecified EB priors preserve FPR control but lose the MSE advantage—in the pooled case producing higher MSE than uncorrected GST—confirming that the Tier 3 estimation advantage is inseparable from corpus quality.

Setting C sweeps corpus size and quality. Representative corpora converge to oracle-level FPR and FDR by roughly 30–50 experiments, tracking the Storey approximation FDR ≈ 0.12 at these parameters. Winner-selected corpora remain at FPR ≈ 0.20 and elevated FDR at every corpus size; pooled programs stay persistently inflated. Notably, the biased corpora stop faster than oracle—the same weak evidence standard that produces their inflated error rates.

Setting D evaluates expected regret per decision as a function of shipping cost $P(B > A) > \gamma$4. At $P(B > A) > \gamma$5, expected-loss stopping has the lowest regret. But it ships roughly half of all experiments including most nulls, and BF stopping at even a loose calibration overtakes it once $P(B > A) > \gamma$6 exceeds about 0.003—roughly 2% of the mean absolute effect among non-null experiments. Expected-loss stopping minimizes regret only when shipping a null-effect variant is nearly free.

Limitations and open questions

The paper is candid about several constraints. Key results assume known $P(B > A) > \gamma$7, extending only asymptotically to estimated variance. The near-optimality of BF thresholds under asymmetric costs is supported numerically but not quantified. The FNR bound excludes paths reaching $P(B > A) > \gamma$8 without crossing either boundary, and is loose for small effects near zero. The Setting C DGP is all-or-nothing at the experiment level, so per-metric and experiment-level non-null rates coincide—a simplification acknowledged not to hold generally. Robustness mechanisms for misspecified priors ($P(B > A) > \gamma$9-contamination, mixture fallbacks, e-value structural robustness) preserve Tier 2 control, but the paper leaves open how FDR calibration degrades and recovers under each mechanism in the sequential setting. Finally, the exchangeability assumption underlying the EB prior fails when effect sizes shrink over time or program scope shifts; how to detect and adapt to such drift is not addressed.

Conclusion

This overview demonstrates that Bayesian A/B testing configurations span a wide range of statistical behavior, from exact reproduction of naive peeking to anytime-valid FDR control with calibrated shrinkage. Bayes factor stopping is near-optimal for a broad class of cost functions while simultaneously bounding the false positive rate, making the decision-theoretic and frequentist formulations largely interchangeable parameterizations. The distinctive benefits often attributed to Bayesian methods—multiplicity correction, winner's-curse correction, minimal estimation error—are available only at Tier 3, and require a calibrated empirical Bayes prior fitted to a representative corpus of hundreds of experiments; corpus failure modes defeat calibration regardless of scale. The practical guidance is direct: choose the configuration from the risks the experimentation program needs to control, not from the label attached to the method.