A closed-form sample size correction for always-valid inference with optional stopping

Abstract: Sequential tests that allow continuous monitoring are common in A/B experimentation. Power calculations for these tests require simulations that are hard to scale across many metrics on an experimentation platform. Instead, a common sizing heuristic inflates the fixed-sample size until the marginal rejection probability at the planned endpoint reaches $1-β$. This last-point rule is conservative because always-valid (AV) power is the probability of a boundary crossing at any time during the run, not at the endpoint alone. We give a closed-form correction factor $k^{α, β, t_0}$ expressed in elementary functions and the bivariate normal CDF, where $t_0 = m/n_z$ is the burn-in fraction. The closed-form approximation depends on the boundary only through its value and slope at the planned endpoint and can be evaluated for any smooth concave boundary. We work out three cases: the confidence sequences of Waudby-Smith et al. (2023) and Maharaj et al. (2023), and the mixture sequential probability ratio test of Johari et al. (2022). Setting the total sample size to $k^ \cdot n_z$, where $n_z$ is the fixed-sample size for allocation ratio $r$, hits empirical power within approximately 3 percentage points of target in Gaussian simulations. The correction factor depends on the allocation ratio $r$ only through $t_0 = m/n_z(r)$. We study sensitivity to the burn-in parameter and show that the correction saves 8--20% of the last-point sample budget across the operating range.

• The paper introduces a closed-form correction factor based on tangent-boundary linearization, Brownian first-passage probabilities, and bivariate normal CDFs to estimate sample sizes for always-valid inference.
• The method reaches target power within about 3 percentage points in simulations, reduces sample requirements by roughly 8%–17.5%, and achieves median production savings of 9.5% across 713 Spotify metrics.
• The correction applies to smooth concave boundaries and arbitrary allocation ratios through the burn-in fraction, but users should verify boundary concavity and exercise caution with very low event rates or untested parameter settings.

Motivation and problem statement

Sequential tests that permit continuous monitoring—confidence sequences and mixture sequential probability ratio tests (mSPRTs)—are now standard on large experimentation platforms. Their power calculations, however, resist closed-form treatment: the always-valid (AV) power is the probability of a boundary crossing at any time from burn-in to the planned endpoint, a first-passage problem for which no elementary expression exists for the curved boundaries in practical use. The prevailing workaround is the "last-point rule," which inflates the fixed-sample size $n_z$ until the marginal rejection probability at the endpoint alone reaches $1-\beta$. Because this ignores crossings before the endpoint, it is conservative by construction: at $(\alpha, 1-\beta) = (0.05, 0.80)$ the paper reports empirical power between $0.86$ and $0.88$, i.e., oversizing by seven to nine percentage points.

The paper's contribution is a closed-form correction factor $k^*(\alpha, \beta, t_0)$, where $t_0 = m/n_z$ is the burn-in fraction, such that setting the total sample size to $k^* \cdot n_z$ approximately achieves target AV power without simulation. The factor is expressed entirely in elementary functions and the bivariate normal CDF, requires only two $\Phi_2$ evaluations per bracket-search step, and depends on the boundary only through its value $b(k)$ and slope $1-\beta$0 at the planned endpoint—so it applies to any smooth concave boundary.

Setup: boundaries and rescaled time

The author considers three widely deployed boundaries under a two-sample model with allocation ratio $1-\beta$1, standardised effect $1-\beta$2, and fixed-sample size

$1-\beta$3

WSKR confidence sequence: $1-\beta$4 with $1-\beta$5 from the Robbins–Siegmund limiting distribution (Waudby-Smith et al., 2023). Maharaj confidence sequence: an asymptotic Gaussian-mixture supermartingale boundary with tuning constant $1-\beta$6 involving the Lambert-$1-\beta$7 function. mSPRT: the Johari–Pekelis–Walsh boundary with mixing standard deviation set to the MDE, whose rejection rule coincides with a Bayes factor threshold [johariopre2022].

A key normalisation makes the analysis allocation-agnostic. Defining $1-\beta$8 gives unit variance for every $1-\beta$9; under $(\alpha, 1-\beta) = (0.05, 0.80)$0 at $(\alpha, 1-\beta) = (0.05, 0.80)$1, the rescaled process $(\alpha, 1-\beta) = (0.05, 0.80)$2 converges weakly to Brownian motion with drift $(\alpha, 1-\beta) = (0.05, 0.80)$3 and variance rate one, started at $(\alpha, 1-\beta) = (0.05, 0.80)$4. Consequently $(\alpha, 1-\beta) = (0.05, 0.80)$5 depends on $(\alpha, 1-\beta) = (0.05, 0.80)$6 only through $(\alpha, 1-\beta) = (0.05, 0.80)$7; at equal $(\alpha, 1-\beta) = (0.05, 0.80)$8, the factor is identical across allocation ratios. This is confirmed empirically: simulated power at $(\alpha, 1-\beta) = (0.05, 0.80)$9 and $0.86$0 matches the $0.86$1 values within Monte Carlo error when $0.86$2 is computed at the $0.86$3-specific $0.86$4.

All three boundaries are concave on $0.86$5—proved analytically for WSKR and Maharaj, verified numerically for mSPRT. Concavity is essential: it guarantees the tangent at the endpoint lies above the boundary, making the linearised crossing probability a lower bound on the true crossing probability and hence making the resulting $0.86$6 slightly conservative rather than anti-conservative.

The corrected factor

The derivation follows Siegmund's classical strategy: linearise the boundary by its tangent at the planned endpoint $0.86$7, apply the Bachelier first-passage formula to Brownian motion with drift over the linear surrogate, then integrate over the random initial value $0.86$8 using a Gaussian convolution identity that reduces each term to a bivariate normal CDF. The resulting approximation decomposes into three interpretable pieces:

$0.86$9

where $0.88$0 is the immediate-rejection probability at burn-in under the tangent surrogate, $0.88$1 is the crossing probability during $0.88$2, and $0.88$3 is the Bachelier reflection correction. The corrected factor $0.88$4 is the smallest root of $0.88$5.

At $0.88$6 and $0.88$7, representative factors are $0.88$8 (WSKR), $0.88$9 (Maharaj), and $k^*(\alpha, \beta, t_0)$0 (mSPRT), with savings over the last-point rule of roughly $k^*(\alpha, \beta, t_0)$1–$k^*(\alpha, \beta, t_0)$2. Across the full grid ($k^*(\alpha, \beta, t_0)$3, $k^*(\alpha, \beta, t_0)$4), savings range from about $k^*(\alpha, \beta, t_0)$5 to $k^*(\alpha, \beta, t_0)$6. The author cautions that because the three boundaries arise from different mathematical constructions, the differing factors should not be read as an efficiency ranking.

Two caveats attach to uniqueness and monotonicity. Monotonicity of $k^*(\alpha, \beta, t_0)$7 in $k^*(\alpha, \beta, t_0)$8 does not follow automatically—the tangent surrogate changes with $k^*(\alpha, \beta, t_0)$9, so crossing events are not nested—and uniqueness of the root is a numerical observation on the tested grid, not a theorem. Users applying the formula at untabulated parameters are advised to verify concavity of the mSPRT boundary numerically.

Simulation validation

Monte Carlo validation uses $t_0 = m/n_z$0 replications per cell (MC SE below $t_0 = m/n_z$1) across Gaussian, Bernoulli, and log-normal outcomes, allocation ratios $t_0 = m/n_z$2, and effect sizes $t_0 = m/n_z$3. Type-I error control is verified first with $t_0 = m/n_z$4 null replications; empirical one-sided rejection rates stay below $t_0 = m/n_z$5 for all three boundaries at every grid cell.

Empirical power at the corrected sample size lands close to target: at $t_0 = m/n_z$6, values cluster around $t_0 = m/n_z$7–$t_0 = m/n_z$8 versus $t_0 = m/n_z$9–$k^* \cdot n_z$0 under last-point sizing. On the extended grid—including Maharaj down to $k^* \cdot n_z$1—the closed form hits target power within approximately $k^* \cdot n_z$2 percentage points, with no degradation at extreme parameters, though savings shrink toward the low-$k^* \cdot n_z$3/high-power corner (e.g., $k^* \cdot n_z$4 at $k^* \cdot n_z$5, $k^* \cdot n_z$6). Against the Monte Carlo reference for $k^* \cdot n_z$7 itself, the closed form agrees within $k^* \cdot n_z$8 relative error (median $k^* \cdot n_z$9) for WSKR and $\Phi_2$0 (median $\Phi_2$1) for Maharaj.

Production validation and burn-in sensitivity

On $\Phi_2$2 metrics drawn from the last 283 experiments on Spotify's Confidence platform (Maharaj boundary with Bonferroni corrections for multiple comparisons and metrics), the median saving is $\Phi_2$3 and the mean $\Phi_2$4. This is below the $\Phi_2$5 at the nominal $\Phi_2$6 cell because Bonferroni adjustment pushes effective $\Phi_2$7 and $\Phi_2$8 toward the lower-saving corner of the grid—an honest reconciliation between laboratory and production numbers.

Sensitivity analysis over burn-ins $\Phi_2$9 reveals a practically important trade-off. Decreasing $b(k)$0 widens the WSKR and Maharaj boundaries through the $b(k)$1 term, and this widening more than offsets the longer monitoring window: $b(k)$2 increases as $b(k)$3 decreases. At realistic burn-ins ($b(k)$4), the total budget $b(k)$5 is $b(k)$6–$b(k)$7 the fixed-sample size—considerably more than the factor near $b(k)$8 obtained if monitoring begins only at $b(k)$9. The mSPRT boundary, which does not depend on $1-\beta$00, yields $1-\beta$01 essentially constant (variation under $1-\beta$02 across $1-\beta$03). The implication is that the baseline cost of anytime-valid inference with realistic early monitoring is substantially higher than naive extrapolation from late-start monitoring suggests, even after the correction.

Limitations and open questions

Several limitations are stated plainly. The Gaussian approximation replaces the estimated $1-\beta$04 in the WSKR construction with known $1-\beta$05, moving from the distribution-free guarantee to a CLT-based one; at low base rates this degrades measurably—with fewer than roughly ten expected successes per arm, empirical power falls below target (gap of $1-\beta$06 at $1-\beta$07 and $1-\beta$08 at $1-\beta$09). Concavity of the mSPRT boundary is verified only numerically on the tabulated grid. Uniqueness of the root and monotonicity of $1-\beta$10 remain unproven in general. Finally, the Bernoulli robustness checks involve unequal arm variances under $1-\beta$11, outside the common-variance model on which the theory rests. Whether a rigorous monotonicity proof, a finite-population or non-Gaussian analogue of the correction, or an extension beyond concave boundaries is possible remains open.

Conclusion

This paper replaces simulation-dependent sizing for always-valid sequential tests with a closed-form factor requiring two bivariate normal CDF evaluations, valid for any smooth concave boundary and any allocation ratio through the single parameter $1-\beta$12. It corrects a seven-to-nine-percentage-point conservatism in the last-point rule, saves $1-\beta$13–$1-\beta$14 of the sample budget across the operating range (median $1-\beta$15 in production), and quantifies the previously underappreciated cost of wide burn-in windows: realistic anytime-valid testing requires budgets two to three times the fixed-sample size.