Sequential Multiple Testing (SMT)

• Sequential Multiple Testing (SMT) is a family of inferential procedures that dynamically evaluates multiple hypotheses as data accrues, ensuring control over global error metrics.
• It employs methodologies such as sequential Holm, gap rules, CSPRT, and online adaptive tests to decide when to stop sampling based on evolving evidence.
• SMT frameworks are applied in domains like online data streams, survival analysis, Monte Carlo testing, and high-dimensional sparse recovery, guaranteeing error control (e.g., FWER, FDR) and asymptotic optimality.

Sequential multiple testing (SMT) denotes a class of testing problems in which multiple hypotheses are handled while evidence accrues over time, and the procedure is designed to stop early whenever the observed data are sufficiently informative while still controlling a prescribed global error criterion. In the literature, SMT includes simultaneous testing of multiple sequential data streams, online multiple testing where hypotheses arrive in a stream, multihypothesis ordered testing via sequential likelihood ratios, and sequential allocation of Monte Carlo effort when exact $p$-values are unavailable. The corresponding guarantees are expressed through familywise error rate (FWER), false discovery rate (FDR), false nondiscovery rate (FNR), generalized familywise criteria, generalized misclassification control, or exact agreement with the classification based on ideal $p$-values (Bartroff et al., 2013, Chen et al., 2017, Chen, 2012, Gandy et al., 2012).

1. Principal formulations of SMT

A central formulation treats a fixed family of hypotheses, one per data stream, with data observed sequentially. In this setting a procedure is typically a pair $(T,d)$ or $(T,D)$, where $T$ is a stopping time and the terminal decision assigns each stream to a null or alternative state. Depending on the model, all streams may stop simultaneously, each stream may stop at a different time, or decisions may be made while all streams continue to be monitored. A recurrent structural feature is the genuinely sequential three-way state space: a hypothesis can be rejected, accepted, or kept under sampling (Bartroff et al., 2013, Bartroff et al., 2013, Xing et al., 2023).

A second formulation concerns multiple ordered hypotheses for a common parameter. In the consecutive sequential probability ratio test (CSPRT) framework, the parameter space is partitioned into consecutive intervals,

$$H_0:\theta\in\Theta_0,\quad H_1:\theta\in\Theta_1,\quad \ldots,\quad H_{m-1}:\theta\in\Theta_{m-1},$$

with indifference zones $(\theta_i',\theta_i'')$ between adjacent regions. The procedure monitors consecutive likelihood ratios between neighboring parameter values and stops once the data are consistent with exactly one interval (Chen, 2012).

A third formulation is online multiple testing. Here the hypotheses $\mathcal H_1,\mathcal H_2,\ldots$ arrive sequentially in a stream, and the decision on $\mathcal H_i$ must use only past information. The canonical examples are LORD and LOND, which choose a significance level $\alpha_i$ adaptively from a budget sequence $p$0 satisfying $p$1 (Chen et al., 2017).

A fourth formulation appears when exact $p$2-values are not analytically available and must be approximated by permutation, bootstrap, or other Monte Carlo tests. In MMCTest and QuickMMCTest, the sequential object is not the underlying scientific data stream but the allocation of Monte Carlo samples across hypotheses, with the aim of reproducing the final multiple-testing classification as efficiently as possible (Gandy et al., 2012, Gandy et al., 2014).

Regime	Sequential object	Representative procedures
Fixed family, sequential data streams	Observations in one or more streams	sequential Holm, gap rule, gap-intersection rule, Leap rule
Ordered multihypothesis testing	Consecutive likelihood ratios for adjacent parameter values	CSPRT
Online testing	Arrival of hypotheses and adaptive significance levels	LORD, LOND
Monte Carlo implementation	Allocation of resampling effort across hypotheses	MMCTest, QuickMMCTest

These regimes share the same broad objective—valid multiple decisions under sequential information—but differ sharply in what arrives over time: observations, hypotheses, likelihood evidence, or Monte Carlo samples. This suggests that SMT is best understood as a family of inferential architectures rather than a single procedure.

2. Error criteria and optimality targets

The earliest multistream SMT formulations in the data block emphasize familywise control. In the sequential Holm framework, the goal is

$p$3

where FWEII is the familywise type II error probability, defined as the probability of at least one incorrect acceptance of a false null. The same familywise perspective appears in gap rules and gap-intersection rules for independent streams, where type I and type II familywise errors are controlled simultaneously, often without requiring any assumptions about dependence across streams (Bartroff et al., 2013, Song et al., 2016).

Subsequent work generalizes the admissible error notions. One line controls the probability of at least $p$4 total mistakes, using the generalized mis-classification rate

$p$5

Another line separately controls the probabilities of at least $p$6 false positives and at least $p$7 false negatives. In later synthesis, these criteria appear as generalized misclassification rate (GMR) and generalized familywise error rates (GFWER), alongside FDR/FNR and the known-number-of-signals class $p$8 (Song et al., 2016, Liu et al., 5 Mar 2026).

Online testing shifts the central metric from FWER to FDR and FNR. For the first $p$9 hypotheses, the paper on LORD and LOND defines

$(T,d)$0

and studies the combined risk

$(T,d)$1

In large-scale sequential-data settings, oracle and data-driven lfdr rules are constructed to ensure simultaneous control of FDR and FNR (Chen et al., 2017, Roy et al., 2023).

Recent work makes the asymptotic objective explicit. First-order asymptotic optimality means that the ratio of the expected sample size to the minimal achievable expected sample size converges to one as the error tolerances vanish. Second-order asymptotic optimality is stronger: for every signal configuration, the difference between the expected sample size and the minimal achievable expected sample size remains uniformly bounded as the error tolerance levels tend to zero (Liu et al., 5 Mar 2026).

A different perspective arises in multi-stream sequential change detection. There, the paper proves that any algorithm with finite average run length (ARL) must have a trivial worst-case false detection rate, FWER, PFER, and GER. To resolve this conflict, it introduces error over patience (EOP), for example

$(T,d)$2

This is a direct statement that some classical worst-case Type I metrics are fundamentally incompatible with finite-ARL sequential detection goals (Dandapanthula et al., 7 Jan 2025).

3. Core sequential mechanisms and representative procedures

The most direct extension of Wald’s binary sequential logic to multiple ordered hypotheses is CSPRT. For the discrete-time case the likelihood ratio is

$(T,d)$3

Sampling continues until there exists an index $(T,d)$4 such that

$(T,d)$5

and

$(T,d)$6

The resulting wrong-decision guarantee is explicit: $(T,d)$7 When $(T,d)$8, CSPRT collapses to Wald’s sequential probability ratio test (Chen, 2012).

For multiple binary streams, a prominent family of procedures is based on ordered local log-likelihood ratios. When the exact number of signals is known, the gap rule stops when the $(T,d)$9-th and $(T,D)$0-st largest log-likelihood ratios are separated by a threshold. When only lower and upper bounds $(T,D)$1 are known, the gap-intersection rule combines intersection-style stopping with rank-gap conditions tailored to the boundary cases $(T,D)$2 and $(T,D)$3. Under i.i.d. observations in each stream, these procedures achieve the optimal expected sample size asymptotically, and in the symmetric case the paper states that knowledge of the exact number of signals can roughly halve the expected sample size relative to the case of no prior information (Song et al., 2016).

Generalized error control leads to different stopping structures. The Sum-Intersection rule stops when the sum of the $(T,D)$4 least significant absolute local log-likelihood ratios exceeds a boundary and is asymptotically optimal for controlling the probability of at least $(T,D)$5 mistakes. The Leap rule is designed for simultaneous control of the probabilities of at least $(T,D)$6 false positives and at least $(T,D)$7 false negatives; it does so by taking the earliest stop among a collection of asymmetric Sum-Intersection subrules, thereby “leaping over” a limited number of hard streams (Song et al., 2016).

The sequential Holm procedure extends Holm’s fixed-sample step-down idea to heterogeneous sequential streams. It assumes stream-specific sequential statistics with critical values that satisfy per-stream Type I and Type II error bounds, standardizes them onto a common scale, and then performs a stagewise step-down rule that accepts small standardized statistics and rejects large ones. Its defining property is simultaneous control of type I FWER and type II FWER regardless of the between-stream correlation, even if streams are highly correlated or duplicated exactly (Bartroff et al., 2013). The rejection principle for sequential tests provides a unifying sufficient condition for FWER control through a rejection function $(T,D)$8, combining a monotonicity condition with a global no-false-rejection condition under the true false hypotheses (Bartroff et al., 2013).

Online procedures implement a different mechanism. LORD uses

$(T,D)$9

where $T$0 is the time of the most recent rejection, whereas LOND uses

$T$1

Under independence of the $T$2-values,

$T$3

for both procedures. In the static sparse AGG model, LORD is shown to be as powerful as the batch Benjamini–Hochberg procedure to first asymptotic order (Chen et al., 2017).

Two later extensions refine the same gap-based logic. In the equicorrelated Gaussian means model, dependence is exploited through the decomposition into a common factor and an independent component, so ordered cumulative-sum gaps behave like differences of independent parts; the resulting gap rules remain asymptotically optimal for FWER and for broader multiple-testing metrics bounded above and below by multiples of FWER (Dey et al., 2023). In the asynchronous setting, decisions for different streams are allowed at different times while using data from all streams, and the proposed rule attains the minimum expected decision time in every stream, simultaneously in every signal configuration, under arbitrary prior bounds on the number of signals (Xing et al., 2023).

4. Monte Carlo, resampling, and combination-based implementations

When ideal $T$4-values $T$5 are not analytically available, MMCTest provides a safe sequential implementation of a monotone multiple testing procedure $T$6. It maintains nested confidence intervals $T$7 for each $T$8, forms the definitely rejected set

$T$9

and the possibly rejected set

$$H_0:\theta\in\Theta_0,\quad H_1:\theta\in\Theta_1,\quad \ldots,\quad H_{m-1}:\theta\in\Theta_{m-1},$$0

Only undecided hypotheses in $$H_0:\theta\in\Theta_0,\quad H_1:\theta\in\Theta_1,\quad \ldots,\quad H_{m-1}:\theta\in\Theta_{m-1},$$1 receive additional Monte Carlo samples. If interrupted, the algorithm returns three sets: rejected hypotheses, non-rejected hypotheses, and undecided hypotheses. Under joint confidence coverage at level $$H_0:\theta\in\Theta_0,\quad H_1:\theta\in\Theta_1,\quad \ldots,\quad H_{m-1}:\theta\in\Theta_{m-1},$$2, MMCTest yields, with arbitrarily high probability, the same classification as the ideal multiple testing procedure applied to the true $$H_0:\theta\in\Theta_0,\quad H_1:\theta\in\Theta_1,\quad \ldots,\quad H_{m-1}:\theta\in\Theta_{m-1},$$3 (Gandy et al., 2012).

QuickMMCTest uses a different criterion: decision stability under posterior resampling. For hypothesis $$H_0:\theta\in\Theta_0,\quad H_1:\theta\in\Theta_1,\quad \ldots,\quad H_{m-1}:\theta\in\Theta_{m-1},$$4, after observing $$H_0:\theta\in\Theta_0,\quad H_1:\theta\in\Theta_1,\quad \ldots,\quad H_{m-1}:\theta\in\Theta_{m-1},$$5 exceedances among $$H_0:\theta\in\Theta_0,\quad H_1:\theta\in\Theta_1,\quad \ldots,\quad H_{m-1}:\theta\in\Theta_{m-1},$$6 Monte Carlo samples, it assigns the Beta posterior

$$H_0:\theta\in\Theta_0,\quad H_1:\theta\in\Theta_1,\quad \ldots,\quad H_{m-1}:\theta\in\Theta_{m-1},$$7

After repeated posterior draws and applications of the multiple testing rule $$H_0:\theta\in\Theta_0,\quad H_1:\theta\in\Theta_1,\quad \ldots,\quad H_{m-1}:\theta\in\Theta_{m-1},$$8, it defines the stability weight

$$H_0:\theta\in\Theta_0,\quad H_1:\theta\in\Theta_1,\quad \ldots,\quad H_{m-1}:\theta\in\Theta_{m-1},$$9

and allocates new Monte Carlo samples proportionally to these weights. The recommended terminal rule rejects $(\theta_i',\theta_i'')$0 when the empirical posterior rejection probability $(\theta_i',\theta_i'')$1 exceeds $(\theta_i',\theta_i'')$2. The paper explicitly contrasts this approach with MMCTest: QuickMMCTest emphasizes reproducibility, power, and efficiency, but does not provide the same exact correctness guarantee (Gandy et al., 2014).

Combination-based SMT addresses dependence by repeatedly testing nested subsets of ordered $(\theta_i',\theta_i'')$3-values. The sequential Cauchy combination test (SCC) orders

$(\theta_i',\theta_i'')$4

and, at stage $(\theta_i',\theta_i'')$5, tests the nested global null

$(\theta_i',\theta_i'')$6

using the stage-$(\theta_i',\theta_i'')$7 Cauchy statistic

$(\theta_i',\theta_i'')$8

Its main guarantee is strong familywise error rate control in the asymptotic small-$(\theta_i',\theta_i'')$9 sense,

$\mathcal H_1,\mathcal H_2,\ldots$0

under Assumption 1 and, when $\mathcal H_1,\mathcal H_2,\ldots$1 diverges, Assumption 2 (Bouamara et al., 2023).

A further large-scale implementation uses local false discovery rate boundaries that shrink the continue-sampling region as the sample size increases. The oracle local index of significance is

$\mathcal H_1,\mathcal H_2,\ldots$2

and sampling stops when the adaptive lower and upper cutoffs cross, equivalently when $\mathcal H_1,\mathcal H_2,\ldots$3. Under the two-group mixture model, the same rule becomes an lfdr procedure, and the data-driven version achieves asymptotic simultaneous control of FDR and FNR as $\mathcal H_1,\mathcal H_2,\ldots$4. The stopping times are proper with probability $\mathcal H_1,\mathcal H_2,\ldots$5 for all finite $\mathcal H_1,\mathcal H_2,\ldots$6 and converge to a finite constant in the large-scale regime (Roy et al., 2023).

5. Specialized domains and domain-specific variants

In high-dimensional sparse recovery, SMT appears as sequential thresholding. The problem is exact recovery of a sparse support $\mathcal H_1,\mathcal H_2,\ldots$7 from $\mathcal H_1,\mathcal H_2,\ldots$8 components, each observed sequentially. The procedure repeatedly measures only the currently active components, computes a local log-likelihood ratio statistic, and removes about half of the null components at each pass by thresholding at the median under the null. With $\mathcal H_1,\mathcal H_2,\ldots$9 passes, false positives vanish because each null survives a stage with probability $\mathcal H_i$0, and under the stated condition on the minimum signal statistic the procedure is reliable. The paper’s central message is that non-sequential recovery scales with $\mathcal H_i$1, whereas sequential thresholding scales with $\mathcal H_i$2 or $\mathcal H_i$3, which can be exponentially better in the dimension (Malloy et al., 2011).

In survival analysis for sequential multiple assignment randomized trials (SMARTs), the “sequential” object is the multistage treatment regime. The generalized logrank-type test compares the survival distributions associated with a set of embedded regimes $\mathcal H_i$4, under the null

$\mathcal H_i$5

Because regimes are counterfactual and may share early treatment paths, a naive logrank test is not adequate. The proposed method builds inverse-probability-weighted score equations with regime-consistency indicators $\mathcal H_i$6, treatment probability products $\mathcal H_i$7, and weights $\mathcal H_i$8, and produces a quadratic-form test statistic whose null distribution is asymptotically $\mathcal H_i$9 with degrees of freedom equal to $\alpha_i$0. The framework handles any number of stages $\alpha_i$1, covariate adjustment, estimated treatment probabilities, and even observational studies under the stated identifiability assumptions (Tsiatis et al., 2024).

In multi-stream sequential change detection, the paper combines e-detectors with e-Benjamini–Hochberg and e-Bonferroni/e-Holm analogues. The detector values $\alpha_i$2 are ranked at each time, and decisions are made by thresholding them in BH-, Bonferroni-, or Holm-style. The principal guarantee is uniform control of EOP over all stopping times; for example, under constant $\alpha_i$3,

$\alpha_i$4

for e-d-BH, and analogous EOP bounds hold for e-d-Bonferroni and e-d-Holm. If one instead chooses $\alpha_i$5, then uniform classical Type I error control is recovered, but only at the price of infinite ARL (Dandapanthula et al., 7 Jan 2025).

SMT also appears in model selection problems. For estimating the number of communities in assortative stochastic block models, the procedure sequentially increases a candidate $\alpha_i$6, forms block adjacency matrices, computes blockwise spectral test statistics based on the second eigenvalue and Tracy–Widom calibration, and accepts the first $\alpha_i$7 such that all selected blocks are Erdős–Rényi according to a multiple-testing threshold. Under balancedness, consistent community recovery when $\alpha_i$8 is known, and the stated sparsity regimes, the final estimator $\alpha_i$9 is consistent (Jha et al., 21 Jul 2025).

6. Prior information, weighting, and higher-order asymptotics

Prior structural information materially changes optimal SMT design. When the number of signals is known exactly, the relevant information rate in the expected sample size lower bound becomes $p$00, and the gap rule attains it. When only bounds $p$01 are known, the optimal first-order sample size depends on whether the true configuration is interior or on a boundary. In the symmetric case with $p$02 and $p$03, the paper states that knowing the exact number of signals can roughly halve the expected sample size relative to the no-prior-information case (Song et al., 2016).

Weighted procedures show that such structural information can be injected directly into the evidence process. The weighted log-likelihood ratio is

$p$04

and the Weighted Gap and Weighted Gap-Intersection procedures rank these weighted statistics rather than the unweighted log-likelihood ratios. The error bound for the Weighted Gap procedure is

$p$05

and first-order asymptotic optimality is preserved. The paper emphasizes that weighting changes only lower-order terms in expected stopping time, even though it may improve or worsen finite-sample behavior (Bose et al., 10 Nov 2025).

A still broader extension allows multiple hypotheses per stream and arbitrary prior information on the full hypothesis configuration. The reliable class then requires control of every type-$p$06 familywise error probability,

$p$07

The optimal lower bound depends on

$p$08

and the proposed procedure stops when the maximum likelihood configuration is separated from every minimal admissible alternative. In the multihypothesis setting the closest alternatives may be cycles or chains rather than single-stream relabelings, and the simulations in the paper show that omitting pairwise or cyclic stopping constraints can destroy error control (Xing, 30 May 2026).

Recent asymptotic theory sharpens the first-order picture. The second-order framework proves that several procedures already known to be first-order optimal—Sum-Intersection, Leap, Intersection, and Gap—are in fact second-order optimal: for every signal configuration, the excess expected sample size over the infimum remains $p$09 as the error levels vanish. In the symmetric case, the minimal achievable expected sample size has the expansion

$p$10

so the second-order correction comes from a boundary-crossing problem for a multidimensional random walk (Liu et al., 5 Mar 2026).

Taken together, these results show that SMT is organized by three interacting axes: the information flow over time, the admissible global error criterion, and the structural information available before sampling begins. A plausible implication is that no single sequential multiple testing procedure can be canonical across all regimes; the method that is optimal for ordered parameter intervals, online FDR, generalized familywise control, safe Monte Carlo implementation, or finite-ARL change detection is determined by the geometry of the error constraint and the way evidence becomes available.