Sequential Hypothesis Testing

• Sequential hypothesis testing is a statistical framework that analyzes data sequentially to enable early stopping when sufficient evidence is reached.
• It adaptively controls error rates, such as the familywise error rate, by generalizing classical methods like Wald’s SPRT to multiple and correlated data streams.
• Its applications in clinical trials and online experiments deliver significant sample efficiency and robust error management in practical settings.

Sequential hypothesis testing is a statistical framework in which samples are collected and analyzed in a sequential manner, allowing for the possibility of early stopping when sufficient evidence is accrued for either hypothesis. This methodology provides an adaptive approach to controlling error probabilities and expected sample size, often yielding substantial efficiency gains over fixed-sample testing paradigms. Sequential procedures generalize the classical paradigm introduced by Wald’s Sequential Probability Ratio Test (SPRT) to more complex scenarios, including multiple hypothesis testing, composite alternatives, and correlated or heterogeneous data streams. The framework is rooted in rigorous error control metrics such as familywise error rates (FWER), and is characterized by procedures that guarantee strong control over both type I and type II errors under arbitrary dependence structures between data streams (Bartroff et al., 2013).

1. Sequential Testing Principles and Error Metrics

Sequential hypothesis testing is defined by the sequential acquisition and analysis of data streams, where each stream corresponds to a distinct experiment or hypothesis. For $k$ hypotheses indexed by $i=1,\dots,k$, the classical goal is to test

$$H_i: \theta_i \in \Theta_i^0 \quad \text{vs.} \quad G_i: \theta_i \in \Theta_i^1$$

with $\Theta_i^0$ and $\Theta_i^1$ disjoint, for each stream. The overall metric of statistical validity is typically cast in terms of the familywise error rate: $$\text{Type I FWER}: \quad \text{FWER}_I(\theta) = \mathbb{P}_\theta(\exists i \in T(\theta): H_i \text{ incorrectly rejected})$$

$$\text{Type II FWER}: \quad \text{FWER}_{II}(\theta) = \mathbb{P}_\theta(\exists i \in F(\theta): H_i \text{ incorrectly accepted})$$

with $T(\theta)$ the set of true nulls and $F(\theta)$ false nulls. Control over both type I (false positive) and type II (false negative) errors is often required simultaneously, with target levels $\alpha, \beta \in (0,1)$ (Bartroff et al., 2013).

2. The Sequential Holm Procedure

The sequential Holm procedure generalizes Holm’s step-down approach to the sequential setting. At each stage, active hypotheses are tested using calibrated sequential statistics $A_i(n)$ with critical boundaries $A_{i,s} < B_{i,s}$. These boundaries are carefully selected so that for each stream $i$ and stage $s$,

$$\sup_{\theta_i \in \Theta_i^0} \mathbb{P}_{\theta_i}(A_i(n) \ge B_{i,s} \text{ at some } n; A_i(n') > A_{i,s} \ \forall n'<n) \le \alpha/(k-s+1)$$

$$\sup_{\theta_i \in \Theta_i^1} \mathbb{P}_{\theta_i}(A_i(n) \le A_{i,s} \text{ at some } n; A_i(n') < B_{i,s} \ \forall n'<n) \le \beta/(k-s+1)$$

The algorithm iteratively samples all active streams, updating standardized statistics $\widetilde{T}_i(n)$ via a monotone transform. Acceptance and rejection thresholds are set in a manner that reflects current stage counts, yielding a step-down update:

• Upon crossing a lower boundary, accept hypotheses with minimum statistics.
• Upon crossing an upper boundary, reject hypotheses with maximum statistics. The procedure continues until all hypotheses are resolved; no hypothesis can be both accepted and rejected in a single stage (Bartroff et al., 2013).

3. Theoretical Properties and Strong Error Control

Bartroff and Song establish that the sequential Holm procedure ensures

$$\sup_\theta \text{FWER}_I(\theta) \le \alpha \qquad \sup_\theta \text{FWER}_{II}(\theta) \le \beta$$

with no assumptions regarding the dependence structure between streams. These properties are proved by leveraging marginal error control within each stream and using union bounds, unaffected by cross-stream correlations or joint distributions. This guarantees robust familywise error control even in the presence of highly correlated, or even duplicated, data streams—a critical requirement for complex experimental designs in scientific, industrial, or clinical contexts (Bartroff et al., 2013).

4. Efficiency: Sample Size and Comparison to Classical Methods

Empirical studies compare the sequential Holm procedure (SH) to:

• Fixed-sample Holm (FH): using predetermined sample sizes,
• Sequential Bonferroni (SB): running parallel SPRTs at $\alpha/k, \beta/k$ per stream,
• Intersection schemes (IS): e.g., De & Baron (2012).

In independent Bernoulli streams ($p_i \le 0.4$ vs. $p_i \ge 0.6$), SH demonstrates average total sample size reductions of $\approx 58\%$ over FH and $\approx 15\%$ over SB, with empirical FWERs that remain close to target levels. In correlated normal-mean scenarios, SH achieves $30\text{–}50\%$ reduction in sample size compared to both FH and SB, with FWERs within $1\text{–}2$ percentage points of nominal levels while alternative methods are overly conservative (Bartroff et al., 2013).

5. Implementation: Single-Stream Statistics, Boundary Construction, and Practical Guidelines

For each stream:

• Simple-vs-simple hypotheses utilize the SPRT log-likelihood ratio

$$A_i(n) = \sum_{j=1}^n \log \frac{g_i(X_{i,j})}{h_i(X_{i,j})}$$

with critical values constructed via Wald’s approximations.

• Composite hypotheses (e.g., unknown variances) employ sequential generalized likelihood-ratio statistics, with boundaries computed via Monte Carlo or normal approximations.
• Standardization of statistics allows for streamwise comparability and efficient implementation across heterogenous data types.

Boundary spending and group sequential protocols are supported by allowing time-dependent boundaries $A_{i,s}(n)$ and $B_{i,s}(n)$. No modeling of between-stream correlation is required—control is achieved via marginal test design. Pseudocode for the sequential Holm procedure prescribes precomputation of boundaries, initialized hypothesis sets, and iterative sampling until decision points are reached (Bartroff et al., 2013).

6. Extensions, Generalizations, and Limitations

Sequential hypothesis testing has been extended to control generalized error rates (e.g., $\gamma$-FDP, $k$-FWER), accommodate group-sequential or truncated sampling, and adaptively handle complex, highly correlated data structures (Bartroff, 2014). Theoretical results demonstrate strong error control and sample efficiency under arbitrary dependence and flexible test statistic design. The approach remains constrained by the need for streamwise error-controlling statistics; handling of hierarchical or networked hypotheses may require additional structure.

7. Significance and Application

Sequential hypothesis testing, particularly as instantiated by the sequential Holm procedure, provides a statistically optimal, computationally feasible, and universally valid framework for multiple hypothesis testing with streaming data. Its type I and II FWER guarantees, sample efficiency, and robustness to correlated data make it an essential methodology for researchers and practitioners in fields where early decisions and error control are critical. The paradigm, with its extensions, embodies current statistical best practice for high-dimensional, large-scale, and online experimental regimes (Bartroff et al., 2013).